-- Logs begin at Sun 2022-10-02 00:10:32 UTC, end at Sun 2022-10-02 01:37:03 UTC. -- Oct 02 00:40:15.849307 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 02 00:40:15.859940 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 02 00:40:15.861974 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Oct 2 00:40:15 2022] *** Oct 02 00:40:15.861974 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 02 00:40:15.861974 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 02 00:40:15.861974 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: nodename: dv-cinder-master-3-iscsi Oct 02 00:40:15.861974 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: machine: x86_64 Oct 02 00:40:15.861974 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: clock source: unix Oct 02 00:40:15.861974 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: pcre jit disabled Oct 02 00:40:15.861974 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: detected number of CPU cores: 4 Oct 02 00:40:15.861974 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: current working directory: / Oct 02 00:40:15.861974 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: detected binary path: /usr/bin/uwsgi-core Oct 02 00:40:15.862634 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: your processes number limit is 47680 Oct 02 00:40:15.862634 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: your memory page size is 4096 bytes Oct 02 00:40:15.862634 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: detected max file descriptor number: 2048 Oct 02 00:40:15.862634 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: lock engine: pthread robust mutexes Oct 02 00:40:15.862634 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: thunder lock: enabled Oct 02 00:40:15.862634 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 02 00:40:15.862634 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 02 00:40:15.901391 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: Python main interpreter initialized at 0x5565da3ff9c0 Oct 02 00:40:15.901391 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: python threads support enabled Oct 02 00:40:15.901564 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: your server socket listen backlog is limited to 100 connections Oct 02 00:40:15.901564 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: your mercy for graceful operations on workers is 90 seconds Oct 02 00:40:15.901698 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: mapped 403077 bytes (393 KB) for 2 cores Oct 02 00:40:15.901804 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: *** Operational MODE: preforking *** Oct 02 00:40:15.902093 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: *** uWSGI is running in multiple interpreter mode *** Oct 02 00:40:15.902093 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: spawned uWSGI master process (pid: 81423) Oct 02 00:40:15.902263 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 02 00:40:15.903356 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: spawned uWSGI worker 1 (pid: 81432, cores: 1) Oct 02 00:40:15.903356 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: spawned uWSGI worker 2 (pid: 81433, cores: 1) Oct 02 00:40:15.903356 dv-cinder-master-3-iscsi devstack@keystone.service[81423]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 02 00:40:20.170715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 02 00:40:20.170715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: from cryptography.utils import int_from_bytes Oct 02 00:40:20.171550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 02 00:40:20.171550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: from cryptography.utils import int_from_bytes Oct 02 00:40:20.172176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 02 00:40:20.172176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: from cryptography.utils import int_from_bytes Oct 02 00:40:20.172621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 02 00:40:20.172621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: from cryptography.utils import int_from_bytes Oct 02 00:40:20.220506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 02 00:40:20.220694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 02 00:40:20.220883 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] command line args: [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 02 00:40:20.221150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 02 00:40:20.221326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ================================================================================ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 02 00:40:20.221576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] admin_token = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.221819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config_dir = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.222006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.222113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 02 00:40:20.222213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] config_source = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.222290 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 02 00:40:20.222390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] debug = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.222478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] command line args: [] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 02 00:40:20.222583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.222855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 02 00:40:20.222925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ================================================================================ {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 02 00:40:20.222988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.223139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.223245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] admin_token = **** {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.223342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.223440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] config_dir = [] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.223534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.223638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.223742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] config_source = [] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.223837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.223935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] debug = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.224037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_config_append = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.224136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.224268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.224268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_dir = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.224450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.224589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.224690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.224783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.224891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.224977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.225091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.225209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.225284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] list_limit = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.225284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] log_config_append = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.225475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.225550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.225663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.225744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] log_dir = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.225820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.225944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] log_file = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.226024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.226120 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.226193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.226308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.226393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.226500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.226593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.226749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.226834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.226950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.227037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.227116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.227231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.227312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.227430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.227556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.227699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.227788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.228016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.228176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.228255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.228378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] publish_errors = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.228454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.228568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.228703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.228781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.228934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.229020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.229173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.229256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.229399 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.229627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] standard_threads = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.229703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.229851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.229938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] publish_errors = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_journal = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_json = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_stderr = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] use_syslog = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.230994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.231144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.231270 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] standard_threads = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.231392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.231477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.231624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.231702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.231805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.231891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.231969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] use_journal = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.232096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.232179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] use_json = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.232306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] use_stderr = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.232388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.232514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] use_syslog = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.232590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.232703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 02 00:40:20.232787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.232958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.233035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.233196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.233275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.233982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.234086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.234244 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.234320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.234481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.234566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.234724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.234800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.234951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.235027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.235193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.235271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.235436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.235522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.235672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.235756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.235918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.235996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.236152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.236235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.236387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.236463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.236618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.236702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.236867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.237121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.237197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.237368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.237518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.237593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.237662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.237825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.237901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.238049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.238158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.238297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.238375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.238527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.238604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.238772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.238850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.239014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.239091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.239240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.239322 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.239466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.239550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.239706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.239784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.239949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.240025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.240181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.240265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.240414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.240496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.240637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.240714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.240867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.240945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.241104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.241221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.241296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.241366 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.241519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.241598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.241741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.241818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.241965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.242041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.242218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.242304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.242494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.242607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.242730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.242838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.242989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.243064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.243182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.243285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.243412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.243516 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.243643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.243745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.243873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.243987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.244108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.244210 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.244359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.244440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.244589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.244666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.244825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.244930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.245056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.245161 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.245292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.245395 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.245513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.245621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.245761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.245874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.245989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.246126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.246259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.246383 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.246469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.246577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.246698 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.246773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.246921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.247003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.247143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.247223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.247372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.247488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.247598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.247688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.247849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.247938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.248079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.248309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.248546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.248632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.248776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.248858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.249028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.249109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.249814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.249901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.250054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.250182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.250328 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.250439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.250569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.250653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.250807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.250880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.251039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.251115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.251272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.251357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.251502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.251578 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.251731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.251809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.251977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.252065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.252189 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.252267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.252417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.252495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.252659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.252744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.252895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.252971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.253138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.253216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.253370 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.253453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.253601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.253679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.253835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.253911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.254094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.254187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.254315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.254392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.254545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.254621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.254769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.254919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.254995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.255065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.255204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.255280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.255431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.255518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.255654 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.255732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.255892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.255970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.256119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.256197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.256361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.256446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.256598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.256679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.256826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.256903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.257058 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.257137 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.257294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.257371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.257517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.external = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.257592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.257742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.257821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.257982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.258087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.258254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.258366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.258493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.258573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.password = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.258715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] auth.token = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.258794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.258949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.259036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.259179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.259259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.259420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.259499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] auth.external = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.259646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.259723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.259886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.259965 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.260126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.260212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.260353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.260431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] auth.password = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.260582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.260660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] auth.token = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.260819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.260898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.261054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.261132 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.261294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.261371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.261525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.261604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.261760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.261837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.261988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.262126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.262245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.262325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.262482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] credential.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.262819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.262974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.263051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.263222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.263302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.263469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.263545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.263712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.263790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.263957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.264037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.264191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.264269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.264434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.264513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.264703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.264783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.264938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.265015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.265178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.265254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.265418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.265498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.265644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.265722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.265889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.265967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.266140 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.266223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.266389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.266470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.266619 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.266706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] federation.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.266868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.266947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.267100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.267180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.267342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.267428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.267571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.267648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.267809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.267886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.268046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.268130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.268274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.268351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.268502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.268580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.268742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.268891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.268967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.269034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.269185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.269263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.269426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.269513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.269672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.269750 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.269900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.269977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.270149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.270243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.270377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.270458 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.270616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.270694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.270845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.270929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.271085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.271164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.271331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.271416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.271572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.271734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.271816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.271887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.272020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.272100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.272253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.272339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.272485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.272563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.272716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.272941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.273017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.273168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.273248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.273402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.273482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.273636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.273714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.273877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.273957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.274127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.274210 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.274372 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.274449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.274613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.274694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.274844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.274922 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.275085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.275163 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.275318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.275397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.275541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.275618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.275778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.275856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.276004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.276083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.276235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.276313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.276466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.276552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.276702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.276781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.276933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.277011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.277178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.277262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.277425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.277504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.277663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.277740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.277902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.277988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.278155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.278234 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.278394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.278472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.278630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.278720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.278873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.278951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.279105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.279183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.279336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.279447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.279561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.279701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.279779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.279953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.280031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.280186 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.280266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.280447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.280525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.280699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.280781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.280950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.281030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.281176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.281253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.281401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.281517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.281631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.281718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.281859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.281937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.282107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.282189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.282334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.282415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.282574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.282652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.282813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.282891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.283054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.283134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.283294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.283373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.283528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.283605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.283750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.283829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.283984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.284061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.284211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.284289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.284441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.284520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.284668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.284745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.284895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.284971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.285134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.285214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.285375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.285453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.285605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.285683 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.285849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.285928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.286079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.286158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.286332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.286409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.286560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.286640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.286789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.286867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.287021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.287098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.287257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.287342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.287489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.287567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.287716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.287795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.287948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.288034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.288182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.288264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.288424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.288501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.288664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.288743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.288896 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.288974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.289130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.289206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.289367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.289447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.289594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.289671 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.289834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.289912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.290097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.290180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.290318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.290395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.290547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.290624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] resource.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.290791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.290872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.291017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.291095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.291258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.291409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.291487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.291559 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.291730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.291809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.291963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.292048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.292199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.292277 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.292434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.292511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.292669 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.292754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.292901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] role.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.292979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.293124 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.293200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] role.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.293363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.293449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.293601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.293680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.293841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.293920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.294131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.294228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.294350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.294484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.294597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.294714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.294842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.294958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.295088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.295200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.295335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.295440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.295567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.295685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.295813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.296447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.296594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.296699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.296841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.296927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.297086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.297165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.297381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.297458 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.297606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.297686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.297830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.297933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.298090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.298216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.298343 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.298465 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.298584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.298711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.298814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.298947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.299034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.299201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.299276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.299452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.299530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.299689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.299774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.299919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.299997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.300150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.300226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.300378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.300466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.300630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.300707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.300861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.300938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.301101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.301187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.301338 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.301415 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.301574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.301652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.301802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.301887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.302035 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] token.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.302131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.302294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.302373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.302532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.302620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.302770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.302848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.303017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.303102 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.303253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.303333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.303487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.303566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.303731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.303808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.303976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.304054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.304213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.304292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.304444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.304522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.304677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.304755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.304906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.304985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.305136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.305214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.305366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.305444 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.305596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.305676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.305837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.305924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.306094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.306177 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.306322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.306400 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.connection = **** {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.306551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.306631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.306789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.306867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.307018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.307096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.307255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.307335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.307480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.307558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.307790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.307940 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.308020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.308092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81433) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 02 00:40:20.308163 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.308408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.308643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.308886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.309120 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.309367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.309602 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.309845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.310091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.310335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.310569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 02 00:40:20.310740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81432) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 02 00:40:20.337327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.337556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.337780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.339227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.339461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.339689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.346669 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.348534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.374126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.374355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.376518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.376749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.427352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.428271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.475973 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.476201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.477543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.477784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.632308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.632308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.632689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.632766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.632930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.633144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.634727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.635091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.635355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.635617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.635978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.636366 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.656792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.657001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.657194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.657370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.657527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.657612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.657730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.657810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.657993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.658143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.658330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.658635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.658935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.659200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.669197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.672196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.682697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.682929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.683241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.683366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.683576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.683807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.684014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81433) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.685785 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.686027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.686430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.687213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.687548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.687907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.688220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81432) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 02 00:40:20.701889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.702132 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 02 00:40:20.703052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.703254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 02 00:40:20.703540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.703791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.704048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.704302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.705230 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.705560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 02 00:40:20.706165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.706413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 02 00:40:20.706778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.707170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.707499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.707851 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.713867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.714172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.714446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.714733 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.717617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.717927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.718238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.718527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.721696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.724608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.724895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.725180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.725465 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.725546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.725751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.726016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.726313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.726591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.726862 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.727136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.727404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.728403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.728692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.728981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.729270 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.729559 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.729827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.730125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.730408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.730683 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.730960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.731231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.750560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.752026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.752310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.754452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.755928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.756214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.756970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.757254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.757534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.760893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.761182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.761470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.764200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.764488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.768688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.768981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.770765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.771055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.771339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.771623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.771900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.772176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.772446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.772717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.774674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.774973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.775264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.775552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.775834 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.776115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.776390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.776663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.788750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.789031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.792880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.793009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.793210 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.796227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.796527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.797164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.800413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.800714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.800837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.803980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.804945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.807215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.807524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.807811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.808078 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.808209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.808375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.808662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.811421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.812261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.812554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.812846 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.813130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.813421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.824947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.825239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.825507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.825770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.829223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.829514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.829786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.830051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.832710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.834278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.835801 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.836079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.836919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.838489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.838848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.839135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.839427 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.839712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.840021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.840146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.840284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.843502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.843790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.844079 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.844362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.844755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.851735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.852030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.852318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.852611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.852993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.853284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.853567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.855935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.856226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.856513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.856800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.857187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.857480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.857764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.868988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.871932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.873238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.875416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.876710 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.877028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.878731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.879150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.879709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.879962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.881409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.882948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.883188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.883610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.890801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.891115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.892263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.893405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.895210 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.895615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.895908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.896191 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.898100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.898649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.899061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.899458 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.904054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.904351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.904647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.911606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81432) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.911894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.912180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.912464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.912746 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.912894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.913037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.913168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.913294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.913418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.913570 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.913847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.914156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.914433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.914705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.914984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.915258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81432) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.920457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81433) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 02 00:40:20.920981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.921436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.921734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.922013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.922397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.922836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.923305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.923587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.923868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.924152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.924423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.924689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.925004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81433) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 02 00:40:20.950085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5565da3ff9c0 pid: 81432 (default app) Oct 02 00:40:20.953958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198fd850-0b74-4879-b9b2-de1778362cea None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:20.954267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198fd850-0b74-4879-b9b2-de1778362cea None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:20.954539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198fd850-0b74-4879-b9b2-de1778362cea None None] PATH_INFO: `/v3` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:20.958691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1/1] 10.222.0.25 () {50 vars in 791 bytes} [Sun Oct 2 00:40:20 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:20.960092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x5565da3ff9c0 pid: 81433 (default app) Oct 02 00:40:27.730786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ad342-335a-4a6f-ba0d-180e7174586d None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:27.731303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ad342-335a-4a6f-ba0d-180e7174586d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:27.731303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ad342-335a-4a6f-ba0d-180e7174586d None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:27.732609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 2/2] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:40:27.740514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:27.741077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:27.741077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:27.817503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: INFO dbcounter [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] Registered counter for database keystone Oct 02 00:40:27.836283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG oslo_db.sqlalchemy.engines [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81433) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 02 00:40:27.842307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writer thread running {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 02 00:40:28.017826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG passlib.handlers.bcrypt [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81433) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 02 00:40:28.018236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG passlib.handlers.bcrypt [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81433) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 02 00:40:28.076009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:28.078169 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:28.201150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1/3] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:27 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 464 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 02 00:40:28.634194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880ad342-335a-4a6f-ba0d-180e7174586d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:28.703896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: INFO dbcounter [None req-880ad342-335a-4a6f-ba0d-180e7174586d None None] Registered counter for database keystone Oct 02 00:40:28.724593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG oslo_db.sqlalchemy.engines [None req-880ad342-335a-4a6f-ba0d-180e7174586d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81432) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 02 00:40:28.730614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writer thread running {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 02 00:40:28.852305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880ad342-335a-4a6f-ba0d-180e7174586d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:28.856607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9a8398-f68c-4c32-be19-ec8b47e4439f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:28.857222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a8398-f68c-4c32-be19-ec8b47e4439f admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:28.857500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a8398-f68c-4c32-be19-ec8b47e4439f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:28.858124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9a8398-f68c-4c32-be19-ec8b47e4439f admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:28.866095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9a8398-f68c-4c32-be19-ec8b47e4439f admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:28.917300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: WARNING keystone.common.rbac_enforcer.enforcer [None req-7d9a8398-f68c-4c32-be19-ec8b47e4439f admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 02 00:40:28.917584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9a8398-f68c-4c32-be19-ec8b47e4439f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:28.927420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 3/4] 10.222.0.25 () {58 vars in 1324 bytes} [Sun Oct 2 00:40:28 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 02 00:40:28.933319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:28.949490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e348208d-82ad-4c82-984c-f88d3d24d408 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:28.953765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbb1d36-8bd7-4eb7-b4dd-c576b87b1bd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:28.955020 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb1d36-8bd7-4eb7-b4dd-c576b87b1bd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:28.955298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb1d36-8bd7-4eb7-b4dd-c576b87b1bd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:28.955576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb1d36-8bd7-4eb7-b4dd-c576b87b1bd2 admin admin] PATH_INFO: `/v3/projects/2ea30d4949414e4cafefc43b9507a1d0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:28.956992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbb1d36-8bd7-4eb7-b4dd-c576b87b1bd2 admin admin] RBAC: Authorizing `identity:get_project(project_id=2ea30d4949414e4cafefc43b9507a1d0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:29.009147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: WARNING keystone.common.rbac_enforcer.enforcer [None req-0fbb1d36-8bd7-4eb7-b4dd-c576b87b1bd2 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 02 00:40:29.009491 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbb1d36-8bd7-4eb7-b4dd-c576b87b1bd2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:29.014870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 2/5] 10.222.0.25 () {56 vars in 1207 bytes} [Sun Oct 2 00:40:28 2022] GET /identity/v3/projects/2ea30d4949414e4cafefc43b9507a1d0 => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:31.135397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cfcd97-c819-4b08-bd76-12ccb6c57f14 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:31.135940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cfcd97-c819-4b08-bd76-12ccb6c57f14 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:31.135940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5cfcd97-c819-4b08-bd76-12ccb6c57f14 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:31.140844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 4/6] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:31 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:31.146278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11df1e9-65a7-462c-9cdf-8068536f78ec None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:31.146687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11df1e9-65a7-462c-9cdf-8068536f78ec None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:31.146821 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11df1e9-65a7-462c-9cdf-8068536f78ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:31.166154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e11df1e9-65a7-462c-9cdf-8068536f78ec None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:31.167638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e11df1e9-65a7-462c-9cdf-8068536f78ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:31.175214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 3/7] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:31 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 02 00:40:31.447202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cfcd97-c819-4b08-bd76-12ccb6c57f14 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:31.459903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5cfcd97-c819-4b08-bd76-12ccb6c57f14 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:31.464007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de771fad-f5e1-493f-bfce-3cd5edea91c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:31.464597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de771fad-f5e1-493f-bfce-3cd5edea91c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:31.464872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de771fad-f5e1-493f-bfce-3cd5edea91c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:31.465145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de771fad-f5e1-493f-bfce-3cd5edea91c2 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:31.473176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de771fad-f5e1-493f-bfce-3cd5edea91c2 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:31.474963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de771fad-f5e1-493f-bfce-3cd5edea91c2 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:31.483360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 5/8] 10.222.0.25 () {58 vars in 1324 bytes} [Sun Oct 2 00:40:31 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 00:40:31.488486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11df1e9-65a7-462c-9cdf-8068536f78ec None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:31.497277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11df1e9-65a7-462c-9cdf-8068536f78ec None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:31.501456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1687915d-c6fa-4143-b541-09615cd5de81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:31.502084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1687915d-c6fa-4143-b541-09615cd5de81 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:31.502367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1687915d-c6fa-4143-b541-09615cd5de81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:31.502636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1687915d-c6fa-4143-b541-09615cd5de81 admin admin] PATH_INFO: `/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:31.503914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1687915d-c6fa-4143-b541-09615cd5de81 admin admin] RBAC: Authorizing `identity:get_user(user_id=b3c5f8b9f24f40b99d0e28c5d77526fa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:31.505712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1687915d-c6fa-4143-b541-09615cd5de81 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:31.507233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 4/9] 10.222.0.25 () {56 vars in 1198 bytes} [Sun Oct 2 00:40:31 2022] GET /identity/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa => generated 247 bytes in 19 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:33.694537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4770ff-7dda-4c9e-8a5e-2f7077495074 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:33.695081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4770ff-7dda-4c9e-8a5e-2f7077495074 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:33.695081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4770ff-7dda-4c9e-8a5e-2f7077495074 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:33.696391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 6/10] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:33.701708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d632af-fe1d-468d-ae62-a64a6a05482d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:33.702116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d632af-fe1d-468d-ae62-a64a6a05482d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:33.702410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d632af-fe1d-468d-ae62-a64a6a05482d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:33.720190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b2d632af-fe1d-468d-ae62-a64a6a05482d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:33.721387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b2d632af-fe1d-468d-ae62-a64a6a05482d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:33.736373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a775a4ef-796c-497a-9005-dcdff493cbaf None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:33.736652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a775a4ef-796c-497a-9005-dcdff493cbaf None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:33.736921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a775a4ef-796c-497a-9005-dcdff493cbaf None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:33.738325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 7/11] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:40:33.743238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e50485c-a432-460f-9086-63de37eafe06 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:33.743520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e50485c-a432-460f-9086-63de37eafe06 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:33.743794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e50485c-a432-460f-9086-63de37eafe06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:33.784995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG passlib.handlers.bcrypt [None req-4e50485c-a432-460f-9086-63de37eafe06 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81432) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 02 00:40:33.785387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG passlib.handlers.bcrypt [None req-4e50485c-a432-460f-9086-63de37eafe06 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81432) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 02 00:40:33.800755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 5/12] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:33.834891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4e50485c-a432-460f-9086-63de37eafe06 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:33.836442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4e50485c-a432-460f-9086-63de37eafe06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:33.936970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 8/13] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:40:34.066710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d632af-fe1d-468d-ae62-a64a6a05482d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:34.091386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:34.172295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d632af-fe1d-468d-ae62-a64a6a05482d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:34.204046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142fc9a0-9865-44d3-b97c-4967914db54a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:34.204224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e50485c-a432-460f-9086-63de37eafe06 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:34.204676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142fc9a0-9865-44d3-b97c-4967914db54a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:34.204951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142fc9a0-9865-44d3-b97c-4967914db54a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:34.205223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142fc9a0-9865-44d3-b97c-4967914db54a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:34.216364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142fc9a0-9865-44d3-b97c-4967914db54a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-142fc9a0-9865-44d3-b97c-4967914db54a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:34.225574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:34.226509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:40:34.227327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 6/14] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:40:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:34.232914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142fc9a0-9865-44d3-b97c-4967914db54a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:34.304417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e50485c-a432-460f-9086-63de37eafe06 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:34.306623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:34.312184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:34.331457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142fc9a0-9865-44d3-b97c-4967914db54a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:34.342228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc28fb0-33fc-4c1d-b7f1-ddd61c66e43d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:34.343094 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc28fb0-33fc-4c1d-b7f1-ddd61c66e43d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:34.343374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc28fb0-33fc-4c1d-b7f1-ddd61c66e43d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:34.343643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc28fb0-33fc-4c1d-b7f1-ddd61c66e43d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:34.364820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f9cf10-a147-4115-8b33-9a6c0e6e88fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:34.365609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f9cf10-a147-4115-8b33-9a6c0e6e88fc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:34.365892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f9cf10-a147-4115-8b33-9a6c0e6e88fc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:34.366186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f9cf10-a147-4115-8b33-9a6c0e6e88fc None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:34.366850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f9cf10-a147-4115-8b33-9a6c0e6e88fc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:34.367143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f9cf10-a147-4115-8b33-9a6c0e6e88fc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:34.368545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f9cf10-a147-4115-8b33-9a6c0e6e88fc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:34.379993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 7/15] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:40:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:34.385380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f9cf10-a147-4115-8b33-9a6c0e6e88fc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:34.411704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc28fb0-33fc-4c1d-b7f1-ddd61c66e43d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:34.413238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc28fb0-33fc-4c1d-b7f1-ddd61c66e43d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:34.483310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f9cf10-a147-4115-8b33-9a6c0e6e88fc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:34.509786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 9/16] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:40:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:40:34.515589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc28fb0-33fc-4c1d-b7f1-ddd61c66e43d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:34.515898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d71fce-dc7a-46ba-92e8-0d5bbabf0633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:34.516522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d71fce-dc7a-46ba-92e8-0d5bbabf0633 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:34.516798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d71fce-dc7a-46ba-92e8-0d5bbabf0633 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:34.517071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d71fce-dc7a-46ba-92e8-0d5bbabf0633 None admin] PATH_INFO: `/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:34.518229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d71fce-dc7a-46ba-92e8-0d5bbabf0633 None admin] RBAC: Authorizing `identity:get_user(user_id=b3c5f8b9f24f40b99d0e28c5d77526fa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:34.518347 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:34.520133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d71fce-dc7a-46ba-92e8-0d5bbabf0633 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:34.521837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 8/17] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:34 2022] GET /identity/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa => generated 247 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:34.527046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d71fce-dc7a-46ba-92e8-0d5bbabf0633 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:34.541052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:34.614836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc28fb0-33fc-4c1d-b7f1-ddd61c66e43d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:34.624803 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d71fce-dc7a-46ba-92e8-0d5bbabf0633 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:34.647692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795b909e-6b42-492c-bde5-9725867e442c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:34.648531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b909e-6b42-492c-bde5-9725867e442c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:34.648810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b909e-6b42-492c-bde5-9725867e442c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:34.649078 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795b909e-6b42-492c-bde5-9725867e442c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:34.657830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c8bc89-c28c-4055-b165-305d4b2930e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:34.658482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c8bc89-c28c-4055-b165-305d4b2930e1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:34.658762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c8bc89-c28c-4055-b165-305d4b2930e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:34.659033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c8bc89-c28c-4055-b165-305d4b2930e1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:34.660138 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c8bc89-c28c-4055-b165-305d4b2930e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:34.661567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c8bc89-c28c-4055-b165-305d4b2930e1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:34.663117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 9/18] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:34.667717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795b909e-6b42-492c-bde5-9725867e442c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:34.668743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c8bc89-c28c-4055-b165-305d4b2930e1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:34.669177 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795b909e-6b42-492c-bde5-9725867e442c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-795b909e-6b42-492c-bde5-9725867e442c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:40:34.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:34.689028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:40:34.690083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:40:34.690083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:40:34.690083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:40:34.690083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 10/19] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:40:34.694935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795b909e-6b42-492c-bde5-9725867e442c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:34.747637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:34.751056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:34.768161 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c8bc89-c28c-4055-b165-305d4b2930e1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:34.795701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795b909e-6b42-492c-bde5-9725867e442c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:34.800895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d0bc98-0d34-4303-a9a8-d850be7d51f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:34.801526 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d0bc98-0d34-4303-a9a8-d850be7d51f5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:34.801805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d0bc98-0d34-4303-a9a8-d850be7d51f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:34.802091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d0bc98-0d34-4303-a9a8-d850be7d51f5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:34.802822 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d0bc98-0d34-4303-a9a8-d850be7d51f5 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.domain.id=default, user.id=b3c5f8b9f24f40b99d0e28c5d77526fa) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:34.803131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d0bc98-0d34-4303-a9a8-d850be7d51f5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:34.804593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d0bc98-0d34-4303-a9a8-d850be7d51f5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:34.811700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 10/20] 10.222.0.25 () {56 vars in 1322 bytes} [Sun Oct 2 00:40:34 2022] GET /identity/v3/role_assignments?user.id=b3c5f8b9f24f40b99d0e28c5d77526fa&scope.domain.id=default&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 233 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:34.830559 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce606f4f-d074-4baf-b2fb-e86fd3d760d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:34.831245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce606f4f-d074-4baf-b2fb-e86fd3d760d1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:34.831564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce606f4f-d074-4baf-b2fb-e86fd3d760d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:34.831857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce606f4f-d074-4baf-b2fb-e86fd3d760d1 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:34.832614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce606f4f-d074-4baf-b2fb-e86fd3d760d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:34.832958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce606f4f-d074-4baf-b2fb-e86fd3d760d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:34.834512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce606f4f-d074-4baf-b2fb-e86fd3d760d1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:34.855611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 11/21] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:40:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:37.439490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df599e3-c26c-468b-93b8-7d6424c9fb2f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:37.439490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df599e3-c26c-468b-93b8-7d6424c9fb2f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:37.439490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df599e3-c26c-468b-93b8-7d6424c9fb2f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:37.439490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 11/22] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:37.444854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b75e37-ead5-4705-a6db-9e9f7f462d31 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:37.445623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b75e37-ead5-4705-a6db-9e9f7f462d31 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:37.446201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b75e37-ead5-4705-a6db-9e9f7f462d31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:37.474215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e6b75e37-ead5-4705-a6db-9e9f7f462d31 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:37.475719 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e6b75e37-ead5-4705-a6db-9e9f7f462d31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:37.560476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 12/23] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:37.603393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4d29fd-cb25-4344-a99b-92566d14ff78 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:37.604013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4d29fd-cb25-4344-a99b-92566d14ff78 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:37.604495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4d29fd-cb25-4344-a99b-92566d14ff78 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:37.605993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 12/24] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:37.611594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd40007-54bd-48e5-a93c-5a29afc54197 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:37.612034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd40007-54bd-48e5-a93c-5a29afc54197 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:37.612465 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd40007-54bd-48e5-a93c-5a29afc54197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:37.642544 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-cfd40007-54bd-48e5-a93c-5a29afc54197 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:37.644033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cfd40007-54bd-48e5-a93c-5a29afc54197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:37.680135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4912b8-2bae-4fdd-8da0-2a23e0e1e41e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:37.680653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4912b8-2bae-4fdd-8da0-2a23e0e1e41e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:37.681081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4912b8-2bae-4fdd-8da0-2a23e0e1e41e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:37.682557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 13/25] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:40:37.687797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5d2a9-7b8f-4168-a57a-32c7a3889928 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:37.688250 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5d2a9-7b8f-4168-a57a-32c7a3889928 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:37.688665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e5d2a9-7b8f-4168-a57a-32c7a3889928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:37.717448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-34e5d2a9-7b8f-4168-a57a-32c7a3889928 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:37.730495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-34e5d2a9-7b8f-4168-a57a-32c7a3889928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:37.736714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:37.789480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 13/26] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:37.790614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7e8aec-2034-48fd-a35e-511a0f7ed4b1 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:37.790894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7e8aec-2034-48fd-a35e-511a0f7ed4b1 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:37.791166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7e8aec-2034-48fd-a35e-511a0f7ed4b1 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:37.792386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 14/27] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:37.799292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acaa200-f21f-4c8a-abf7-79da6ba72027 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:37.799573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acaa200-f21f-4c8a-abf7-79da6ba72027 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:37.799841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acaa200-f21f-4c8a-abf7-79da6ba72027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:37.839220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8acaa200-f21f-4c8a-abf7-79da6ba72027 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:37.840422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8acaa200-f21f-4c8a-abf7-79da6ba72027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:37.884475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 14/28] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:40:37.925824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e5d2a9-7b8f-4168-a57a-32c7a3889928 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:37.983155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 15/29] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:38.036178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:38.069956 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e5d2a9-7b8f-4168-a57a-32c7a3889928 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:38.117058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179c12f0-6eff-474e-9a64-f393ea4e17c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:38.117672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179c12f0-6eff-474e-9a64-f393ea4e17c6 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:38.117954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179c12f0-6eff-474e-9a64-f393ea4e17c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:38.118259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179c12f0-6eff-474e-9a64-f393ea4e17c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:38.118929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179c12f0-6eff-474e-9a64-f393ea4e17c6 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:38.120463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179c12f0-6eff-474e-9a64-f393ea4e17c6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:38.173947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acaa200-f21f-4c8a-abf7-79da6ba72027 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:38.254114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 15/30] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Oct 2 00:40:37 2022] POST /identity/v3/roles => generated 219 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:40:38.285709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179c12f0-6eff-474e-9a64-f393ea4e17c6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:38.371146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acaa200-f21f-4c8a-abf7-79da6ba72027 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:38.386426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:38.439621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b654cf-de3f-40a7-b8f7-283aa8472317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:38.440370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b654cf-de3f-40a7-b8f7-283aa8472317 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:38.440652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b654cf-de3f-40a7-b8f7-283aa8472317 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:38.440922 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b654cf-de3f-40a7-b8f7-283aa8472317 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:38.470811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b654cf-de3f-40a7-b8f7-283aa8472317 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-a6b654cf-de3f-40a7-b8f7-283aa8472317 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:38.478432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:38.479346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:40:38.480089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 16/31] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:40:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:40:38.486621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b654cf-de3f-40a7-b8f7-283aa8472317 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:38.492701 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179c12f0-6eff-474e-9a64-f393ea4e17c6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:38.540926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d4d9bf-4226-4d23-bafb-78392ee555f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:38.541831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d4d9bf-4226-4d23-bafb-78392ee555f2 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:38.542274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d4d9bf-4226-4d23-bafb-78392ee555f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:38.542736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d4d9bf-4226-4d23-bafb-78392ee555f2 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:38.543604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d4d9bf-4226-4d23-bafb-78392ee555f2 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:38.545856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d4d9bf-4226-4d23-bafb-78392ee555f2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:38.551609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:38.568461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 16/32] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Oct 2 00:40:38 2022] POST /identity/v3/roles => generated 225 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:40:38.569913 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d4d9bf-4226-4d23-bafb-78392ee555f2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:38.634483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b654cf-de3f-40a7-b8f7-283aa8472317 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:38.675634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd41292a-8f88-4e43-ac6e-cb6942aa59fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:38.676389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd41292a-8f88-4e43-ac6e-cb6942aa59fc None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:38.676734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd41292a-8f88-4e43-ac6e-cb6942aa59fc None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:38.677106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd41292a-8f88-4e43-ac6e-cb6942aa59fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:38.690885 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd41292a-8f88-4e43-ac6e-cb6942aa59fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:38.692954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd41292a-8f88-4e43-ac6e-cb6942aa59fc None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-bd41292a-8f88-4e43-ac6e-cb6942aa59fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:40:38.711795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:38.712705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:40:38.713529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:40:38.713529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:40:38.713529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:40:38.713529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 17/33] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:40:38.721144 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd41292a-8f88-4e43-ac6e-cb6942aa59fc None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:38.723986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d4d9bf-4226-4d23-bafb-78392ee555f2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:38.747718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:38.776183 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf5581f-b384-4bf6-809d-429f2c5cbcf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:38.777560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf5581f-b384-4bf6-809d-429f2c5cbcf9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:38.778045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf5581f-b384-4bf6-809d-429f2c5cbcf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:38.779370 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf5581f-b384-4bf6-809d-429f2c5cbcf9 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:38.780253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf5581f-b384-4bf6-809d-429f2c5cbcf9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:38.780985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf5581f-b384-4bf6-809d-429f2c5cbcf9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:38.783410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf5581f-b384-4bf6-809d-429f2c5cbcf9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:38.796369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 17/34] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:40:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:38.801770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf5581f-b384-4bf6-809d-429f2c5cbcf9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:38.845958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd41292a-8f88-4e43-ac6e-cb6942aa59fc None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:38.884023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:38.889639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d3638a-cc35-4152-ac0d-0fb1499e6014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:38.890386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d3638a-cc35-4152-ac0d-0fb1499e6014 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:38.890702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d3638a-cc35-4152-ac0d-0fb1499e6014 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:38.890997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d3638a-cc35-4152-ac0d-0fb1499e6014 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:38.891740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d3638a-cc35-4152-ac0d-0fb1499e6014 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:38.892043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d3638a-cc35-4152-ac0d-0fb1499e6014 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:38.893458 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d3638a-cc35-4152-ac0d-0fb1499e6014 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:38.907380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 18/35] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:40:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:38.916785 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d3638a-cc35-4152-ac0d-0fb1499e6014 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:38.945963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf5581f-b384-4bf6-809d-429f2c5cbcf9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:38.991423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cd14ff-557a-4596-8c40-db62a8de89a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:38.991976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cd14ff-557a-4596-8c40-db62a8de89a8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:38.992350 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cd14ff-557a-4596-8c40-db62a8de89a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:38.992739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98cd14ff-557a-4596-8c40-db62a8de89a8 None admin] PATH_INFO: `/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:38.993968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cd14ff-557a-4596-8c40-db62a8de89a8 None admin] RBAC: Authorizing `identity:get_user(user_id=b3c5f8b9f24f40b99d0e28c5d77526fa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:38.995587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98cd14ff-557a-4596-8c40-db62a8de89a8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:38.997534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 18/36] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:38 2022] GET /identity/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa => generated 247 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:39.002974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cd14ff-557a-4596-8c40-db62a8de89a8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:39.027252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:39.057047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d3638a-cc35-4152-ac0d-0fb1499e6014 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:39.103016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0bffdca-bb15-41de-bebb-847240ff6ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:39.103641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bffdca-bb15-41de-bebb-847240ff6ded None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:39.103920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bffdca-bb15-41de-bebb-847240ff6ded None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:39.104191 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bffdca-bb15-41de-bebb-847240ff6ded None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:39.105087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0bffdca-bb15-41de-bebb-847240ff6ded None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:39.107074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0bffdca-bb15-41de-bebb-847240ff6ded None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:39.124664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98cd14ff-557a-4596-8c40-db62a8de89a8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:39.155104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:39.167767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55256e7-a227-4a58-8a7d-223e4ea70f67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:39.168488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55256e7-a227-4a58-8a7d-223e4ea70f67 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:39.168772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55256e7-a227-4a58-8a7d-223e4ea70f67 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:39.169048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55256e7-a227-4a58-8a7d-223e4ea70f67 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:39.170823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55256e7-a227-4a58-8a7d-223e4ea70f67 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:39.172268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55256e7-a227-4a58-8a7d-223e4ea70f67 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:39.173841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 19/37] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:39.175144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc18677b-6c15-4aef-ad2f-89ec0be72f1c None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:39.175430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc18677b-6c15-4aef-ad2f-89ec0be72f1c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:39.175703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc18677b-6c15-4aef-ad2f-89ec0be72f1c None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:39.176955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 20/38] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:39.180657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc18677b-6c15-4aef-ad2f-89ec0be72f1c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:39.248696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 19/39] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:40:38 2022] POST /identity/v3/projects => generated 301 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:40:39.250242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea28f98d-ac52-49c8-a4c7-fd1092296fea None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:39.250534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea28f98d-ac52-49c8-a4c7-fd1092296fea None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:39.250812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea28f98d-ac52-49c8-a4c7-fd1092296fea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:39.272812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ea28f98d-ac52-49c8-a4c7-fd1092296fea None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:39.274587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ea28f98d-ac52-49c8-a4c7-fd1092296fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:39.356883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 20/40] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:39.435627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc18677b-6c15-4aef-ad2f-89ec0be72f1c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:39.474005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a279d78-add6-457b-9e1d-d4e9bf9cc331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:39.474732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a279d78-add6-457b-9e1d-d4e9bf9cc331 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:39.475028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a279d78-add6-457b-9e1d-d4e9bf9cc331 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:39.475359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a279d78-add6-457b-9e1d-d4e9bf9cc331 None admin] PATH_INFO: `/v3/domains/default/users/b3c5f8b9f24f40b99d0e28c5d77526fa/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:39.477612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a279d78-add6-457b-9e1d-d4e9bf9cc331 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=b3c5f8b9f24f40b99d0e28c5d77526fa, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:39.479271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a279d78-add6-457b-9e1d-d4e9bf9cc331 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:39.500597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 21/41] 10.222.0.25 () {58 vars in 1361 bytes} [Sun Oct 2 00:40:39 2022] PUT /identity/v3/domains/default/users/b3c5f8b9f24f40b99d0e28c5d77526fa/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:40:39.598929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f31c7e-49ae-471c-af98-f82488fda91e None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:39.599449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f31c7e-49ae-471c-af98-f82488fda91e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:39.599575 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f31c7e-49ae-471c-af98-f82488fda91e None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:39.601056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 21/42] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:39.606608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0588aac-3716-42bf-9a16-9b63a31a10eb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:39.607568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0588aac-3716-42bf-9a16-9b63a31a10eb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:39.607795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0588aac-3716-42bf-9a16-9b63a31a10eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:39.626944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c0588aac-3716-42bf-9a16-9b63a31a10eb None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:39.628638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c0588aac-3716-42bf-9a16-9b63a31a10eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:39.642672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f31c7e-49ae-471c-af98-f82488fda91e None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:39.689163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:39.712585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 22/43] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:40:39.712859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:39.756733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f31c7e-49ae-471c-af98-f82488fda91e None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:39.791772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce0e5eb-2a73-462d-8601-f960be75f753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:39.792376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce0e5eb-2a73-462d-8601-f960be75f753 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:39.792654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce0e5eb-2a73-462d-8601-f960be75f753 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:39.792924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce0e5eb-2a73-462d-8601-f960be75f753 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:39.793597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce0e5eb-2a73-462d-8601-f960be75f753 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:39.795145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce0e5eb-2a73-462d-8601-f960be75f753 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:39.811166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 22/44] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Oct 2 00:40:39 2022] POST /identity/v3/roles => generated 223 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:40:39.984978 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0588aac-3716-42bf-9a16-9b63a31a10eb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:40.089344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0588aac-3716-42bf-9a16-9b63a31a10eb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:40.122376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc419dc4-864e-40de-bff7-9291fbee2094 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:40.123024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc419dc4-864e-40de-bff7-9291fbee2094 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:40.123304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc419dc4-864e-40de-bff7-9291fbee2094 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:40.123579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc419dc4-864e-40de-bff7-9291fbee2094 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:40.124676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc419dc4-864e-40de-bff7-9291fbee2094 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:40.126160 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc419dc4-864e-40de-bff7-9291fbee2094 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:40.127708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 23/45] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:40.133383 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce0e5eb-2a73-462d-8601-f960be75f753 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:40.229968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce0e5eb-2a73-462d-8601-f960be75f753 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:40.249629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:40.264455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84b6b94-965f-48b8-8771-3e0049d59e1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:40.265068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84b6b94-965f-48b8-8771-3e0049d59e1f None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:40.265349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84b6b94-965f-48b8-8771-3e0049d59e1f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:40.265625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84b6b94-965f-48b8-8771-3e0049d59e1f None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:40.266321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84b6b94-965f-48b8-8771-3e0049d59e1f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:40.267767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84b6b94-965f-48b8-8771-3e0049d59e1f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:40.295806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 23/46] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:40:40 2022] POST /identity/v3/projects => generated 312 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:40:41.648339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb6db7c-000b-483b-8959-b91f8c6178f0 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:41.648848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb6db7c-000b-483b-8959-b91f8c6178f0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:41.648975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb6db7c-000b-483b-8959-b91f8c6178f0 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:41.650348 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 24/47] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:41.656095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a55177-8f26-4fe4-bf82-7be69c3e5b88 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:41.656462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a55177-8f26-4fe4-bf82-7be69c3e5b88 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:41.656680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a55177-8f26-4fe4-bf82-7be69c3e5b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:41.674778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-69a55177-8f26-4fe4-bf82-7be69c3e5b88 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:41.675952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-69a55177-8f26-4fe4-bf82-7be69c3e5b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:41.738779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 24/48] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:42.023603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb6db7c-000b-483b-8959-b91f8c6178f0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:42.074694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:42.124556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb6db7c-000b-483b-8959-b91f8c6178f0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:42.158447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51fadb9-63a5-4b35-8ca8-37e7b7dd4b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:42.159071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51fadb9-63a5-4b35-8ca8-37e7b7dd4b8d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:42.159349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51fadb9-63a5-4b35-8ca8-37e7b7dd4b8d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:42.159619 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51fadb9-63a5-4b35-8ca8-37e7b7dd4b8d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:42.169744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51fadb9-63a5-4b35-8ca8-37e7b7dd4b8d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b51fadb9-63a5-4b35-8ca8-37e7b7dd4b8d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:42.174623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:42.175615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:40:42.176363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 25/49] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:40:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:42.181451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a55177-8f26-4fe4-bf82-7be69c3e5b88 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:42.277885 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a55177-8f26-4fe4-bf82-7be69c3e5b88 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:42.279314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:42.313999 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fd76fe-f8b0-4a30-8e7c-8aeb3aed48e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:42.314669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fd76fe-f8b0-4a30-8e7c-8aeb3aed48e9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:42.314948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fd76fe-f8b0-4a30-8e7c-8aeb3aed48e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:42.315223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fd76fe-f8b0-4a30-8e7c-8aeb3aed48e9 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:42.315909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fd76fe-f8b0-4a30-8e7c-8aeb3aed48e9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:42.316216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fd76fe-f8b0-4a30-8e7c-8aeb3aed48e9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:42.317655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fd76fe-f8b0-4a30-8e7c-8aeb3aed48e9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:42.331133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 25/50] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:40:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:42.338747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51fadb9-63a5-4b35-8ca8-37e7b7dd4b8d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:42.433374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf90475e-506a-4ffa-b0dc-3e65d61fa68d None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:42.433661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf90475e-506a-4ffa-b0dc-3e65d61fa68d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:42.433944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf90475e-506a-4ffa-b0dc-3e65d61fa68d None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:42.435822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 26/51] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:40:42.438080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51fadb9-63a5-4b35-8ca8-37e7b7dd4b8d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:42.440870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f61e485-b8fb-410f-9feb-52b526ab71e7 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:42.441248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f61e485-b8fb-410f-9feb-52b526ab71e7 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:42.441522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f61e485-b8fb-410f-9feb-52b526ab71e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:42.448721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:42.460439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-6f61e485-b8fb-410f-9feb-52b526ab71e7 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:42.461616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6f61e485-b8fb-410f-9feb-52b526ab71e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:42.472294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82246f-e269-4620-9f19-290765940eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:42.472939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82246f-e269-4620-9f19-290765940eb9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:42.473229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82246f-e269-4620-9f19-290765940eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:42.473512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82246f-e269-4620-9f19-290765940eb9 None admin] PATH_INFO: `/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:42.474698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d82246f-e269-4620-9f19-290765940eb9 None admin] RBAC: Authorizing `identity:get_user(user_id=b3c5f8b9f24f40b99d0e28c5d77526fa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:42.476352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d82246f-e269-4620-9f19-290765940eb9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:42.477906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 26/52] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:42 2022] GET /identity/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:42.483253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82246f-e269-4620-9f19-290765940eb9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:42.585546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82246f-e269-4620-9f19-290765940eb9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:42.619747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a25b8b-b198-462a-bd0b-fb7aece6c01d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:42.620378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a25b8b-b198-462a-bd0b-fb7aece6c01d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:42.620656 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a25b8b-b198-462a-bd0b-fb7aece6c01d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:42.620930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a25b8b-b198-462a-bd0b-fb7aece6c01d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:42.622155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a25b8b-b198-462a-bd0b-fb7aece6c01d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:42.624149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a25b8b-b198-462a-bd0b-fb7aece6c01d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:42.625710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 27/53] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:42.631556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a25b8b-b198-462a-bd0b-fb7aece6c01d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:42.632353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 27/54] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:40:42.677162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:42.729101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a25b8b-b198-462a-bd0b-fb7aece6c01d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:42.762427 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de0983-c294-4b17-bff1-24d6f42a84f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:42.763051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de0983-c294-4b17-bff1-24d6f42a84f1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:42.763330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de0983-c294-4b17-bff1-24d6f42a84f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:42.763599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8de0983-c294-4b17-bff1-24d6f42a84f1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:42.764318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8de0983-c294-4b17-bff1-24d6f42a84f1 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.domain.id=default, user.id=b3c5f8b9f24f40b99d0e28c5d77526fa) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:42.764625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8de0983-c294-4b17-bff1-24d6f42a84f1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:42.766052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8de0983-c294-4b17-bff1-24d6f42a84f1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:42.771864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 28/55] 10.222.0.25 () {56 vars in 1322 bytes} [Sun Oct 2 00:40:42 2022] GET /identity/v3/role_assignments?user.id=b3c5f8b9f24f40b99d0e28c5d77526fa&scope.domain.id=default&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 530 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:42.900853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f61e485-b8fb-410f-9feb-52b526ab71e7 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:42.994322 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:43.001977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f61e485-b8fb-410f-9feb-52b526ab71e7 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:43.037282 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bc4d6f-296d-4408-8e6c-896756ff49a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:43.037911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bc4d6f-296d-4408-8e6c-896756ff49a9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:43.038246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bc4d6f-296d-4408-8e6c-896756ff49a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:43.038520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bc4d6f-296d-4408-8e6c-896756ff49a9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:43.039600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bc4d6f-296d-4408-8e6c-896756ff49a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:43.041026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bc4d6f-296d-4408-8e6c-896756ff49a9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:43.042593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 28/56] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:43.048532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de0983-c294-4b17-bff1-24d6f42a84f1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:43.145422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8de0983-c294-4b17-bff1-24d6f42a84f1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:43.155151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:43.177824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20651e1d-00ec-4aa0-9f62-15551a3eff4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:43.178448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20651e1d-00ec-4aa0-9f62-15551a3eff4b None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:43.178725 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20651e1d-00ec-4aa0-9f62-15551a3eff4b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:43.178997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20651e1d-00ec-4aa0-9f62-15551a3eff4b None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:43.179643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20651e1d-00ec-4aa0-9f62-15551a3eff4b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:43.181096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20651e1d-00ec-4aa0-9f62-15551a3eff4b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:43.208927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 29/57] 10.222.0.25 () {60 vars in 1140 bytes} [Sun Oct 2 00:40:43 2022] POST /identity/v3/projects => generated 298 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:40:45.318732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073f0db6-ea3c-4aba-a78c-3b7bec831225 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:45.319195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073f0db6-ea3c-4aba-a78c-3b7bec831225 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:45.319195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073f0db6-ea3c-4aba-a78c-3b7bec831225 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:45.320547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 29/58] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:45.326015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8721bf-f725-4cc4-b36f-afb23744bdb3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:45.326411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8721bf-f725-4cc4-b36f-afb23744bdb3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:45.326592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8721bf-f725-4cc4-b36f-afb23744bdb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:45.344638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1a8721bf-f725-4cc4-b36f-afb23744bdb3 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:45.345813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1a8721bf-f725-4cc4-b36f-afb23744bdb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:45.409528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 30/59] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:45.677701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073f0db6-ea3c-4aba-a78c-3b7bec831225 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:45.775871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073f0db6-ea3c-4aba-a78c-3b7bec831225 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:45.808010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb71f86-9cd4-4abd-9c10-bbcf40a54918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:45.808645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb71f86-9cd4-4abd-9c10-bbcf40a54918 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:45.808928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb71f86-9cd4-4abd-9c10-bbcf40a54918 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:45.809205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb71f86-9cd4-4abd-9c10-bbcf40a54918 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:45.810970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb71f86-9cd4-4abd-9c10-bbcf40a54918 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:45.812448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb71f86-9cd4-4abd-9c10-bbcf40a54918 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:45.814018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 30/60] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:45.819522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8721bf-f725-4cc4-b36f-afb23744bdb3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:45.906468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:45.915966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8721bf-f725-4cc4-b36f-afb23744bdb3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:45.957212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7472d5-916d-460d-8b79-0250d0f85a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:45.957750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7472d5-916d-460d-8b79-0250d0f85a69 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:45.958031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7472d5-916d-460d-8b79-0250d0f85a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:45.958338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7472d5-916d-460d-8b79-0250d0f85a69 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:45.958993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7472d5-916d-460d-8b79-0250d0f85a69 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:45.960445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7472d5-916d-460d-8b79-0250d0f85a69 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:45.986976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 31/61] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:40:45 2022] POST /identity/v3/users => generated 275 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:40:49.327488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69270307-e503-48d3-a4a7-43c7f125f71a None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:49.327991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b64fed-58af-420f-9b44-4ef06dd64991 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:49.328336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69270307-e503-48d3-a4a7-43c7f125f71a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:49.328436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b64fed-58af-420f-9b44-4ef06dd64991 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:49.328521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69270307-e503-48d3-a4a7-43c7f125f71a None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:49.328602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b64fed-58af-420f-9b44-4ef06dd64991 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:49.328687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 31/62] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:49.328759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 32/63] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:49.332554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d067d32c-bad3-40af-910b-f3a53f5fa6c5 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:49.332676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9ead71-c333-4f7b-9be4-4107232ec224 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:49.332917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9ead71-c333-4f7b-9be4-4107232ec224 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:49.333222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d067d32c-bad3-40af-910b-f3a53f5fa6c5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:49.333328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9ead71-c333-4f7b-9be4-4107232ec224 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:49.333738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d067d32c-bad3-40af-910b-f3a53f5fa6c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:49.359142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5b9ead71-c333-4f7b-9be4-4107232ec224 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:49.360633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5b9ead71-c333-4f7b-9be4-4107232ec224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:49.364044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-d067d32c-bad3-40af-910b-f3a53f5fa6c5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:49.366390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d067d32c-bad3-40af-910b-f3a53f5fa6c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:49.412753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:49.449665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 33/64] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:49.498185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e3292-c06a-48ae-9351-c3941be2756b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:49.498715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e3292-c06a-48ae-9351-c3941be2756b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:49.498816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 32/65] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:40:49.499280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32e3292-c06a-48ae-9351-c3941be2756b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:49.501104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 34/66] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:49.515680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ba3e7e-a8d1-4846-a25e-102fb899e475 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:49.515968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ba3e7e-a8d1-4846-a25e-102fb899e475 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:49.516242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ba3e7e-a8d1-4846-a25e-102fb899e475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:49.543363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f6ba3e7e-a8d1-4846-a25e-102fb899e475 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:49.545053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f6ba3e7e-a8d1-4846-a25e-102fb899e475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:49.639510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 33/67] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:49.800031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13bb28-67d2-4575-83cb-83981575954c None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:49.800645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13bb28-67d2-4575-83cb-83981575954c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:49.801111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b13bb28-67d2-4575-83cb-83981575954c None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:49.802783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 35/68] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:49.808296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f326537-16fe-4236-8c12-ff751a1e4cd3 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:49.808834 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f326537-16fe-4236-8c12-ff751a1e4cd3 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:49.809860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f326537-16fe-4236-8c12-ff751a1e4cd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:49.821086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13bb28-67d2-4575-83cb-83981575954c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:49.847591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9f326537-16fe-4236-8c12-ff751a1e4cd3 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:49.849909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9f326537-16fe-4236-8c12-ff751a1e4cd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:49.936240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 34/69] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:49.937091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f326537-16fe-4236-8c12-ff751a1e4cd3 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:49.942408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:49.974414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:49.977860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b13bb28-67d2-4575-83cb-83981575954c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.039845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e203e5-71f7-4378-9248-081ab3f3e16d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.040494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e203e5-71f7-4378-9248-081ab3f3e16d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.040776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e203e5-71f7-4378-9248-081ab3f3e16d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.041051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e203e5-71f7-4378-9248-081ab3f3e16d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.052898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e203e5-71f7-4378-9248-081ab3f3e16d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-65e203e5-71f7-4378-9248-081ab3f3e16d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:50.057750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:50.058628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:40:50.059901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 36/70] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:40:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:50.061194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc06c7-e6ac-42ea-8029-d044e2d92da5 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.061388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc06c7-e6ac-42ea-8029-d044e2d92da5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.061665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48bc06c7-e6ac-42ea-8029-d044e2d92da5 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.063052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 37/71] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:50.066936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bc06c7-e6ac-42ea-8029-d044e2d92da5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.072435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f326537-16fe-4236-8c12-ff751a1e4cd3 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.108409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7235b80c-ef13-42ae-accb-a6cda7ef0b48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.109048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7235b80c-ef13-42ae-accb-a6cda7ef0b48 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.109331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7235b80c-ef13-42ae-accb-a6cda7ef0b48 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.109608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7235b80c-ef13-42ae-accb-a6cda7ef0b48 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.120047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7235b80c-ef13-42ae-accb-a6cda7ef0b48 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-7235b80c-ef13-42ae-accb-a6cda7ef0b48 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:50.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:50.126479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:40:50.127260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 35/72] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:40:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:50.127874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7235b80c-ef13-42ae-accb-a6cda7ef0b48 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.176660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48bc06c7-e6ac-42ea-8029-d044e2d92da5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.210225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d7799e-cf80-4eb2-8552-f6881adcf954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.210785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d7799e-cf80-4eb2-8552-f6881adcf954 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.211069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d7799e-cf80-4eb2-8552-f6881adcf954 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.211349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d7799e-cf80-4eb2-8552-f6881adcf954 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.215034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:50.218658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:50.222883 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d7799e-cf80-4eb2-8552-f6881adcf954 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b1d7799e-cf80-4eb2-8552-f6881adcf954 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:50.227932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:50.228786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:40:50.229571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 38/73] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:50.230236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9e7543-2389-47b4-9d17-32d9fe6c733c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.230921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9e7543-2389-47b4-9d17-32d9fe6c733c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.230921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9e7543-2389-47b4-9d17-32d9fe6c733c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.239761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7235b80c-ef13-42ae-accb-a6cda7ef0b48 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.250595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ac9e7543-2389-47b4-9d17-32d9fe6c733c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:50.251840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ac9e7543-2389-47b4-9d17-32d9fe6c733c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:50.273295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60025de7-7090-463d-8bce-1f380e6fa7de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.273901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60025de7-7090-463d-8bce-1f380e6fa7de None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.274215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60025de7-7090-463d-8bce-1f380e6fa7de None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.274496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60025de7-7090-463d-8bce-1f380e6fa7de None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.275158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60025de7-7090-463d-8bce-1f380e6fa7de None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:50.275464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60025de7-7090-463d-8bce-1f380e6fa7de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:50.276845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60025de7-7090-463d-8bce-1f380e6fa7de None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:50.285753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 36/74] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:50.290927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60025de7-7090-463d-8bce-1f380e6fa7de None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.317661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 39/75] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:50.334241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9e7543-2389-47b4-9d17-32d9fe6c733c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.389905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60025de7-7090-463d-8bce-1f380e6fa7de None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.414836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9e7543-2389-47b4-9d17-32d9fe6c733c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.424245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07531bf2-6ad4-4f48-bcaa-d7be351f91da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.424902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07531bf2-6ad4-4f48-bcaa-d7be351f91da None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.425190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07531bf2-6ad4-4f48-bcaa-d7be351f91da None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.425475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07531bf2-6ad4-4f48-bcaa-d7be351f91da None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.426178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07531bf2-6ad4-4f48-bcaa-d7be351f91da None admin] RBAC: Adding query filter params (name=member) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:50.426460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07531bf2-6ad4-4f48-bcaa-d7be351f91da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:50.427827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07531bf2-6ad4-4f48-bcaa-d7be351f91da None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:50.436171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 37/76] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:50.436961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07531bf2-6ad4-4f48-bcaa-d7be351f91da None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.444199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:50.448336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:50.458259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b7495c-e2af-4140-b654-856c2a80ff71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.459043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b7495c-e2af-4140-b654-856c2a80ff71 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.459324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b7495c-e2af-4140-b654-856c2a80ff71 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.459603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b7495c-e2af-4140-b654-856c2a80ff71 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.470597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b7495c-e2af-4140-b654-856c2a80ff71 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-77b7495c-e2af-4140-b654-856c2a80ff71 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:50.475487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:50.476522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:40:50.477315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 40/77] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:50.477945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b7495c-e2af-4140-b654-856c2a80ff71 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.548035 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07531bf2-6ad4-4f48-bcaa-d7be351f91da None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.583127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62c66af-c2f2-41ae-b556-9fc92f3dd62f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.583732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62c66af-c2f2-41ae-b556-9fc92f3dd62f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.583895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b7495c-e2af-4140-b654-856c2a80ff71 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.584048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62c66af-c2f2-41ae-b556-9fc92f3dd62f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.584278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62c66af-c2f2-41ae-b556-9fc92f3dd62f None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.584930 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62c66af-c2f2-41ae-b556-9fc92f3dd62f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:50.585354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62c66af-c2f2-41ae-b556-9fc92f3dd62f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:50.586845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62c66af-c2f2-41ae-b556-9fc92f3dd62f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:50.595689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 38/78] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:50.596484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62c66af-c2f2-41ae-b556-9fc92f3dd62f None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.622660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190a3478-e4f7-4351-9ef6-12f43253feca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.623288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a3478-e4f7-4351-9ef6-12f43253feca None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.623584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a3478-e4f7-4351-9ef6-12f43253feca None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.623857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a3478-e4f7-4351-9ef6-12f43253feca None admin] PATH_INFO: `/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.624975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190a3478-e4f7-4351-9ef6-12f43253feca None admin] RBAC: Authorizing `identity:get_user(user_id=b3c5f8b9f24f40b99d0e28c5d77526fa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:50.626462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190a3478-e4f7-4351-9ef6-12f43253feca None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:50.628009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 41/79] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa => generated 247 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:50.628754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190a3478-e4f7-4351-9ef6-12f43253feca None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.664241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:50.685197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:50.698861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62c66af-c2f2-41ae-b556-9fc92f3dd62f None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.727775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190a3478-e4f7-4351-9ef6-12f43253feca None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.731617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8deeb62e-a495-488a-b7dc-3929c962580e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.732738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8deeb62e-a495-488a-b7dc-3929c962580e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.733017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8deeb62e-a495-488a-b7dc-3929c962580e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.733289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8deeb62e-a495-488a-b7dc-3929c962580e None admin] PATH_INFO: `/v3/users/d7ae9bf858ce45f1887ed9fbf2036581` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.747729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8deeb62e-a495-488a-b7dc-3929c962580e None admin] RBAC: Authorizing `identity:get_user(user_id=d7ae9bf858ce45f1887ed9fbf2036581)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:50.749183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8deeb62e-a495-488a-b7dc-3929c962580e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:50.750984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 39/80] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/users/d7ae9bf858ce45f1887ed9fbf2036581 => generated 275 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:50.751560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8deeb62e-a495-488a-b7dc-3929c962580e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.760596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c0acab-f5c9-43c8-8803-8d6becd78720 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.761462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c0acab-f5c9-43c8-8803-8d6becd78720 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.761741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c0acab-f5c9-43c8-8803-8d6becd78720 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.762020 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c0acab-f5c9-43c8-8803-8d6becd78720 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.762754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c0acab-f5c9-43c8-8803-8d6becd78720 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:50.763059 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c0acab-f5c9-43c8-8803-8d6becd78720 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:50.764581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c0acab-f5c9-43c8-8803-8d6becd78720 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:50.777689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 42/81] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:50.777689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c0acab-f5c9-43c8-8803-8d6becd78720 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.854337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8deeb62e-a495-488a-b7dc-3929c962580e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.875730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c0acab-f5c9-43c8-8803-8d6becd78720 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:50.879141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:50.891152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e4c61b-5f2e-4121-ac13-0100202a1395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.891792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e4c61b-5f2e-4121-ac13-0100202a1395 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.892075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e4c61b-5f2e-4121-ac13-0100202a1395 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.892353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e4c61b-5f2e-4121-ac13-0100202a1395 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.892773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:50.893630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e4c61b-5f2e-4121-ac13-0100202a1395 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:50.895257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e4c61b-5f2e-4121-ac13-0100202a1395 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:50.897040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 40/82] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:50.897581 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e4c61b-5f2e-4121-ac13-0100202a1395 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.914664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d60f8bb-b545-4d62-a619-cfc6dea55303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:50.915295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d60f8bb-b545-4d62-a619-cfc6dea55303 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:50.915577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d60f8bb-b545-4d62-a619-cfc6dea55303 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:50.915853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d60f8bb-b545-4d62-a619-cfc6dea55303 None admin] PATH_INFO: `/v3/users/d7ae9bf858ce45f1887ed9fbf2036581` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:50.916991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d60f8bb-b545-4d62-a619-cfc6dea55303 None admin] RBAC: Authorizing `identity:get_user(user_id=d7ae9bf858ce45f1887ed9fbf2036581)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:50.918494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d60f8bb-b545-4d62-a619-cfc6dea55303 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:50.920249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 43/83] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/users/d7ae9bf858ce45f1887ed9fbf2036581 => generated 275 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:50.920862 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d60f8bb-b545-4d62-a619-cfc6dea55303 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:50.996560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e4c61b-5f2e-4121-ac13-0100202a1395 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:51.018004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d60f8bb-b545-4d62-a619-cfc6dea55303 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:51.030049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bd0521-9b9e-479e-95c5-e78790211fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:51.030760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bd0521-9b9e-479e-95c5-e78790211fba None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:51.031075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bd0521-9b9e-479e-95c5-e78790211fba None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:51.031353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bd0521-9b9e-479e-95c5-e78790211fba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:51.053091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a4726f-3fd1-43ae-9b6e-4e1a5b774e86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:51.053693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a4726f-3fd1-43ae-9b6e-4e1a5b774e86 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:51.053971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a4726f-3fd1-43ae-9b6e-4e1a5b774e86 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:51.054285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a4726f-3fd1-43ae-9b6e-4e1a5b774e86 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:51.088099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:51.098023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42bd0521-9b9e-479e-95c5-e78790211fba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:51.099498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42bd0521-9b9e-479e-95c5-e78790211fba None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:51.107748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:51.121049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a4726f-3fd1-43ae-9b6e-4e1a5b774e86 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:51.123210 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a4726f-3fd1-43ae-9b6e-4e1a5b774e86 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:51.198889 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 41/84] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:40:51.204825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bd0521-9b9e-479e-95c5-e78790211fba None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:51.221930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 44/85] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:40:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:40:51.226523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a4726f-3fd1-43ae-9b6e-4e1a5b774e86 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:51.292699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:51.302194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bd0521-9b9e-479e-95c5-e78790211fba None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:51.307846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:51.323507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a4726f-3fd1-43ae-9b6e-4e1a5b774e86 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:51.336478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365962ec-a3ee-4d63-a319-568add39e7bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:51.337098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365962ec-a3ee-4d63-a319-568add39e7bc None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:51.337380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365962ec-a3ee-4d63-a319-568add39e7bc None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:51.337656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365962ec-a3ee-4d63-a319-568add39e7bc None admin] PATH_INFO: `/v3/users/d7ae9bf858ce45f1887ed9fbf2036581` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:51.338788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365962ec-a3ee-4d63-a319-568add39e7bc None admin] RBAC: Authorizing `identity:get_user(user_id=d7ae9bf858ce45f1887ed9fbf2036581)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:51.340206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365962ec-a3ee-4d63-a319-568add39e7bc None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:51.341896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 42/86] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:51 2022] GET /identity/v3/users/d7ae9bf858ce45f1887ed9fbf2036581 => generated 275 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:51.342682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365962ec-a3ee-4d63-a319-568add39e7bc None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:51.360196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba8613-efdc-4328-a734-2049339ec603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:51.360806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba8613-efdc-4328-a734-2049339ec603 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:51.361090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba8613-efdc-4328-a734-2049339ec603 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:51.361368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deba8613-efdc-4328-a734-2049339ec603 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:51.362021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deba8613-efdc-4328-a734-2049339ec603 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:51.363502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deba8613-efdc-4328-a734-2049339ec603 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:51.384269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 45/87] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:40:51 2022] POST /identity/v3/users => generated 289 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:40:51.385143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba8613-efdc-4328-a734-2049339ec603 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:51.459808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365962ec-a3ee-4d63-a319-568add39e7bc None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:51.497629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb99b56-361d-4eeb-8f82-8edf8c0aac1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:51.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb99b56-361d-4eeb-8f82-8edf8c0aac1b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:51.498566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb99b56-361d-4eeb-8f82-8edf8c0aac1b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:51.498839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb99b56-361d-4eeb-8f82-8edf8c0aac1b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:51.513217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deba8613-efdc-4328-a734-2049339ec603 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:51.530429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:51.551517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3ed212-9992-4f2f-a642-b6962594002b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:51.552061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3ed212-9992-4f2f-a642-b6962594002b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:51.552335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3ed212-9992-4f2f-a642-b6962594002b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:51.552612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3ed212-9992-4f2f-a642-b6962594002b None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:51.553746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3ed212-9992-4f2f-a642-b6962594002b None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:51.555304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3ed212-9992-4f2f-a642-b6962594002b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:51.556813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 46/88] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:40:51 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:51.557649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3ed212-9992-4f2f-a642-b6962594002b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:51.576672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb99b56-361d-4eeb-8f82-8edf8c0aac1b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:51.578826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb99b56-361d-4eeb-8f82-8edf8c0aac1b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:51.578941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:51.690206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3ed212-9992-4f2f-a642-b6962594002b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:51.694505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 43/89] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:40:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:40:51.695372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb99b56-361d-4eeb-8f82-8edf8c0aac1b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:51.730758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e67f98-05f4-460f-a4fb-4acbc087b02a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:51.731923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e67f98-05f4-460f-a4fb-4acbc087b02a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:51.732206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e67f98-05f4-460f-a4fb-4acbc087b02a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:51.732484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e67f98-05f4-460f-a4fb-4acbc087b02a None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:51.733613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e67f98-05f4-460f-a4fb-4acbc087b02a None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:51.736839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e67f98-05f4-460f-a4fb-4acbc087b02a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:51.737175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 47/90] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:40:51 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:51.737806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e67f98-05f4-460f-a4fb-4acbc087b02a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:51.767501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:51.811932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb99b56-361d-4eeb-8f82-8edf8c0aac1b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:51.846995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c371a3e7-326d-4977-a130-d630ed323f4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:51.847892 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c371a3e7-326d-4977-a130-d630ed323f4c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:51.848380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c371a3e7-326d-4977-a130-d630ed323f4c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:51.848823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c371a3e7-326d-4977-a130-d630ed323f4c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:51.851082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:51.880850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e67f98-05f4-460f-a4fb-4acbc087b02a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:51.924394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c371a3e7-326d-4977-a130-d630ed323f4c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:51.924761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0db2d9-ea27-4b33-aa8b-b5119578c913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:51.925375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0db2d9-ea27-4b33-aa8b-b5119578c913 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:51.925655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0db2d9-ea27-4b33-aa8b-b5119578c913 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:51.925939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0db2d9-ea27-4b33-aa8b-b5119578c913 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:51.926708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0db2d9-ea27-4b33-aa8b-b5119578c913 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e, user.id=b3c5f8b9f24f40b99d0e28c5d77526fa) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:51.927024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0db2d9-ea27-4b33-aa8b-b5119578c913 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:51.927238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c371a3e7-326d-4977-a130-d630ed323f4c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:51.928468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0db2d9-ea27-4b33-aa8b-b5119578c913 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:51.935437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 48/91] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:40:51 2022] GET /identity/v3/role_assignments?user.id=b3c5f8b9f24f40b99d0e28c5d77526fa&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:51.936062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0db2d9-ea27-4b33-aa8b-b5119578c913 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:51.995409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:52.051687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 44/92] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:40:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:40:52.052779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c371a3e7-326d-4977-a130-d630ed323f4c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:52.080232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0db2d9-ea27-4b33-aa8b-b5119578c913 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:52.127975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6fed0b-3200-4f60-8c28-1edb07264026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:52.128781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fed0b-3200-4f60-8c28-1edb07264026 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:52.129160 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fed0b-3200-4f60-8c28-1edb07264026 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:52.129537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fed0b-3200-4f60-8c28-1edb07264026 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:52.130863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6fed0b-3200-4f60-8c28-1edb07264026 None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:52.132442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6fed0b-3200-4f60-8c28-1edb07264026 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:52.134547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 49/93] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:40:51 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:52.138221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6fed0b-3200-4f60-8c28-1edb07264026 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:52.157605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:52.236663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c371a3e7-326d-4977-a130-d630ed323f4c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:52.274180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6fed0b-3200-4f60-8c28-1edb07264026 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:52.315051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a44a143-792e-4972-a773-f0ef24efb6b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:52.315668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a44a143-792e-4972-a773-f0ef24efb6b4 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:52.315972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a44a143-792e-4972-a773-f0ef24efb6b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:52.316217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a44a143-792e-4972-a773-f0ef24efb6b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:52.316940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a44a143-792e-4972-a773-f0ef24efb6b4 None admin] RBAC: Adding query filter params (role.id=e4fa1b2d8c404336a855b6142fbf25d4, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e, user.id=d7ae9bf858ce45f1887ed9fbf2036581) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:52.317248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a44a143-792e-4972-a773-f0ef24efb6b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:52.318794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a44a143-792e-4972-a773-f0ef24efb6b4 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:52.320562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81324eba-a525-4150-916b-697c514b57a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:52.321487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81324eba-a525-4150-916b-697c514b57a0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:52.321966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81324eba-a525-4150-916b-697c514b57a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:52.322501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81324eba-a525-4150-916b-697c514b57a0 None admin] PATH_INFO: `/v3/projects/449b012a1fc24571bd474f23d9c1f706` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:52.324186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81324eba-a525-4150-916b-697c514b57a0 None admin] RBAC: Authorizing `identity:get_project(project_id=449b012a1fc24571bd474f23d9c1f706)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:52.326388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81324eba-a525-4150-916b-697c514b57a0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:52.328228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 50/94] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:40:52 2022] GET /identity/v3/projects/449b012a1fc24571bd474f23d9c1f706 => generated 312 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:52.328613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 45/95] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:40:52 2022] GET /identity/v3/role_assignments?user.id=d7ae9bf858ce45f1887ed9fbf2036581&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 259 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:52.334392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81324eba-a525-4150-916b-697c514b57a0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:52.335728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a44a143-792e-4972-a773-f0ef24efb6b4 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:52.403827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:52.465534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:52.514847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81324eba-a525-4150-916b-697c514b57a0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:52.577901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547c6c29-8e51-4a33-93b5-60c59545a0a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:52.578710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547c6c29-8e51-4a33-93b5-60c59545a0a5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:52.579000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547c6c29-8e51-4a33-93b5-60c59545a0a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:52.579273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547c6c29-8e51-4a33-93b5-60c59545a0a5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:52.583764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547c6c29-8e51-4a33-93b5-60c59545a0a5 None admin] RBAC: Adding query filter params (role.id=5e19a16e5bfd4c94b1c34ae717ccfc42, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e, user.id=d7ae9bf858ce45f1887ed9fbf2036581) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:52.583764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547c6c29-8e51-4a33-93b5-60c59545a0a5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:52.583764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547c6c29-8e51-4a33-93b5-60c59545a0a5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:52.590455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 51/96] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:40:52 2022] GET /identity/v3/role_assignments?user.id=d7ae9bf858ce45f1887ed9fbf2036581&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 259 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:52.608841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a44a143-792e-4972-a773-f0ef24efb6b4 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:52.649587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9896f63d-f614-4c96-89b4-9a8882fe8089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:52.650829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9896f63d-f614-4c96-89b4-9a8882fe8089 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:52.651304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9896f63d-f614-4c96-89b4-9a8882fe8089 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:52.651767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9896f63d-f614-4c96-89b4-9a8882fe8089 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:52.653375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9896f63d-f614-4c96-89b4-9a8882fe8089 None admin] RBAC: Adding query filter params (role.id=e4fa1b2d8c404336a855b6142fbf25d4, scope.project.id=449b012a1fc24571bd474f23d9c1f706, user.id=d7ae9bf858ce45f1887ed9fbf2036581) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:52.653885 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9896f63d-f614-4c96-89b4-9a8882fe8089 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:52.655583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9896f63d-f614-4c96-89b4-9a8882fe8089 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:52.661844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 46/97] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:40:52 2022] GET /identity/v3/role_assignments?user.id=d7ae9bf858ce45f1887ed9fbf2036581&scope.project.id=449b012a1fc24571bd474f23d9c1f706&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 259 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:54.437042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87fb91d-b8f2-4630-83cc-a5d4213182c7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:54.437548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87fb91d-b8f2-4630-83cc-a5d4213182c7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:54.437548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87fb91d-b8f2-4630-83cc-a5d4213182c7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:54.438920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 52/98] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:54.445252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11e0e6-99a1-4544-9b00-6cc3d9d5b150 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:54.445745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11e0e6-99a1-4544-9b00-6cc3d9d5b150 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:54.445817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b11e0e6-99a1-4544-9b00-6cc3d9d5b150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:54.474204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4b11e0e6-99a1-4544-9b00-6cc3d9d5b150 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:54.475452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4b11e0e6-99a1-4544-9b00-6cc3d9d5b150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:54.550220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf2cbc0-f80c-4be0-8310-4e2fa9abce84 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:54.550569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf2cbc0-f80c-4be0-8310-4e2fa9abce84 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:54.550854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf2cbc0-f80c-4be0-8310-4e2fa9abce84 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:54.552178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 53/99] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:40:54.558101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190d035f-b8b0-4604-9ad1-ab752eb374ac None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:54.558519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190d035f-b8b0-4604-9ad1-ab752eb374ac None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:54.559321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190d035f-b8b0-4604-9ad1-ab752eb374ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:54.567451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 47/100] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:54.593155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-190d035f-b8b0-4604-9ad1-ab752eb374ac None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:54.594384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-190d035f-b8b0-4604-9ad1-ab752eb374ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:54.704839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 54/101] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Oct 02 00:40:54.937736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b11e0e6-99a1-4544-9b00-6cc3d9d5b150 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:54.956182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:55.021818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35891631-134a-4fd9-a4d7-0022eb87f3cb None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:55.022165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35891631-134a-4fd9-a4d7-0022eb87f3cb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:55.022397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35891631-134a-4fd9-a4d7-0022eb87f3cb None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:55.023875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 55/102] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:55.042091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b740c6-80b8-49b6-a86b-bd463d895ced None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:55.042091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b740c6-80b8-49b6-a86b-bd463d895ced None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:55.042091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b740c6-80b8-49b6-a86b-bd463d895ced None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:55.074426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-03b740c6-80b8-49b6-a86b-bd463d895ced None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:55.076061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-03b740c6-80b8-49b6-a86b-bd463d895ced None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:55.080789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b11e0e6-99a1-4544-9b00-6cc3d9d5b150 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:55.139307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa804e1-9f06-4d5d-b508-22ce647f6352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:55.139948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa804e1-9f06-4d5d-b508-22ce647f6352 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:55.140231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa804e1-9f06-4d5d-b508-22ce647f6352 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:55.140502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa804e1-9f06-4d5d-b508-22ce647f6352 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:55.158665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa804e1-9f06-4d5d-b508-22ce647f6352 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-5aa804e1-9f06-4d5d-b508-22ce647f6352 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:55.163609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:55.164475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:40:55.165226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:40:55.170497 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 48/103] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:40:54 2022] GET /identity/v3/roles/reader => generated 84 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:55.175872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa804e1-9f06-4d5d-b508-22ce647f6352 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:55.240313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 56/104] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:40:55.241751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0659d3ec-4192-46a1-a2cf-371d6a9d2d44 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:55.242230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0659d3ec-4192-46a1-a2cf-371d6a9d2d44 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:55.242635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0659d3ec-4192-46a1-a2cf-371d6a9d2d44 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:55.244187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 57/105] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:55.245034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0659d3ec-4192-46a1-a2cf-371d6a9d2d44 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:55.248103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:55.337472 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:55.364115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0659d3ec-4192-46a1-a2cf-371d6a9d2d44 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:55.372145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa804e1-9f06-4d5d-b508-22ce647f6352 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:55.402225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304d96a2-566c-4033-a3d3-31f123a6fadf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:55.403767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304d96a2-566c-4033-a3d3-31f123a6fadf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:55.404349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304d96a2-566c-4033-a3d3-31f123a6fadf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:55.404831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304d96a2-566c-4033-a3d3-31f123a6fadf None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:55.412682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304d96a2-566c-4033-a3d3-31f123a6fadf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:55.419715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304d96a2-566c-4033-a3d3-31f123a6fadf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:55.421294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 58/106] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:55.422474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd08d322-490d-41f4-b184-5fddfedd775f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:55.422754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd08d322-490d-41f4-b184-5fddfedd775f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:55.423030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd08d322-490d-41f4-b184-5fddfedd775f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:55.434839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0be72e2-3a08-4a7a-aba3-c6f50e28a47b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:55.437896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0be72e2-3a08-4a7a-aba3-c6f50e28a47b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:55.438438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0be72e2-3a08-4a7a-aba3-c6f50e28a47b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:55.440610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0be72e2-3a08-4a7a-aba3-c6f50e28a47b None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:55.441509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0be72e2-3a08-4a7a-aba3-c6f50e28a47b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:55.443269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0be72e2-3a08-4a7a-aba3-c6f50e28a47b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:55.444880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0be72e2-3a08-4a7a-aba3-c6f50e28a47b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:55.464503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fd08d322-490d-41f4-b184-5fddfedd775f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:55.466463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 49/107] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:40:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:55.467805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0be72e2-3a08-4a7a-aba3-c6f50e28a47b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:55.468674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fd08d322-490d-41f4-b184-5fddfedd775f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:55.599568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:55.604380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 59/108] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:55.605232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd08d322-490d-41f4-b184-5fddfedd775f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:55.632817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0be72e2-3a08-4a7a-aba3-c6f50e28a47b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:55.688187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2486b6a-a2ff-4426-a73d-ab7a7eee940a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:55.688814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2486b6a-a2ff-4426-a73d-ab7a7eee940a None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:55.689093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2486b6a-a2ff-4426-a73d-ab7a7eee940a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:55.689372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2486b6a-a2ff-4426-a73d-ab7a7eee940a None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:55.690025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2486b6a-a2ff-4426-a73d-ab7a7eee940a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:55.692159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2486b6a-a2ff-4426-a73d-ab7a7eee940a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:55.713921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:55.725947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd08d322-490d-41f4-b184-5fddfedd775f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:55.741068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 50/109] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:40:55 2022] POST /identity/v3/projects => generated 302 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:40:55.741918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2486b6a-a2ff-4426-a73d-ab7a7eee940a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:55.816424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a34ae75-a3d3-4db3-945b-c480ff239f3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:55.817063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a34ae75-a3d3-4db3-945b-c480ff239f3a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:55.817338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a34ae75-a3d3-4db3-945b-c480ff239f3a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:55.817614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a34ae75-a3d3-4db3-945b-c480ff239f3a None admin] PATH_INFO: `/v3/users/d7c226f9fa6c4d79867811e016a8115e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:55.845572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a34ae75-a3d3-4db3-945b-c480ff239f3a None admin] RBAC: Authorizing `identity:get_user(user_id=d7c226f9fa6c4d79867811e016a8115e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:55.847108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a34ae75-a3d3-4db3-945b-c480ff239f3a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:55.850784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 60/110] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:55 2022] GET /identity/v3/users/d7c226f9fa6c4d79867811e016a8115e => generated 289 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:55.856353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a34ae75-a3d3-4db3-945b-c480ff239f3a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:55.957157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2486b6a-a2ff-4426-a73d-ab7a7eee940a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:55.966499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:56.026555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7157dde-723e-4f01-88b8-2301d88479e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:56.028067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7157dde-723e-4f01-88b8-2301d88479e7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.029435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7157dde-723e-4f01-88b8-2301d88479e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.030156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7157dde-723e-4f01-88b8-2301d88479e7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.044185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a34ae75-a3d3-4db3-945b-c480ff239f3a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:56.047756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7157dde-723e-4f01-88b8-2301d88479e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-b7157dde-723e-4f01-88b8-2301d88479e7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:56.053229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:56.054169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:40:56.055135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:40:56.057132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 51/111] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:40:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:56.058099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7157dde-723e-4f01-88b8-2301d88479e7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:56.089632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3099d192-cdb5-471d-8c86-37a672004622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:56.090305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3099d192-cdb5-471d-8c86-37a672004622 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.090582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3099d192-cdb5-471d-8c86-37a672004622 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.091099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3099d192-cdb5-471d-8c86-37a672004622 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.126111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:56.192578 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3099d192-cdb5-471d-8c86-37a672004622 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:56.193924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7157dde-723e-4f01-88b8-2301d88479e7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:56.194750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3099d192-cdb5-471d-8c86-37a672004622 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:56.238429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba261e4-f61e-4ad5-91e5-bf8e9d0e56be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:56.239077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba261e4-f61e-4ad5-91e5-bf8e9d0e56be None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.239362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba261e4-f61e-4ad5-91e5-bf8e9d0e56be None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.239641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba261e4-f61e-4ad5-91e5-bf8e9d0e56be None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.251767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba261e4-f61e-4ad5-91e5-bf8e9d0e56be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-bba261e4-f61e-4ad5-91e5-bf8e9d0e56be None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:56.257591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:56.258483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:40:56.259248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 52/112] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:40:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:56.265022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c7b59a-dad5-461a-ade2-28dbd497ce65 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.265022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c7b59a-dad5-461a-ade2-28dbd497ce65 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.265022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c7b59a-dad5-461a-ade2-28dbd497ce65 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.265022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 53/113] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:56.265022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c7b59a-dad5-461a-ade2-28dbd497ce65 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:56.280957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:56.328530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 61/114] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:40:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:40:56.329690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3099d192-cdb5-471d-8c86-37a672004622 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:56.374962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c7b59a-dad5-461a-ade2-28dbd497ce65 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:56.388701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:56.411293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451d516d-1da5-4924-97e8-18efa5a37c70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:56.411946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451d516d-1da5-4924-97e8-18efa5a37c70 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.412696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451d516d-1da5-4924-97e8-18efa5a37c70 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.412987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451d516d-1da5-4924-97e8-18efa5a37c70 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.413668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451d516d-1da5-4924-97e8-18efa5a37c70 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:56.414321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451d516d-1da5-4924-97e8-18efa5a37c70 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:56.415720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451d516d-1da5-4924-97e8-18efa5a37c70 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:56.431786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 54/115] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:40:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:56.431786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd13392-4f39-4bff-99fd-03a58dfa7e44 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.431786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd13392-4f39-4bff-99fd-03a58dfa7e44 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.431786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd13392-4f39-4bff-99fd-03a58dfa7e44 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.431786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 55/116] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:56 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:56.431786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef22ee9-780c-49a7-8967-5bffd520ae01 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.431786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef22ee9-780c-49a7-8967-5bffd520ae01 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.432270 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef22ee9-780c-49a7-8967-5bffd520ae01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.443311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3099d192-cdb5-471d-8c86-37a672004622 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:56.449969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-eef22ee9-780c-49a7-8967-5bffd520ae01 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:56.451616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-eef22ee9-780c-49a7-8967-5bffd520ae01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:56.477264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76859a4-3736-4c65-845c-27eb66bf1c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:56.477877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76859a4-3736-4c65-845c-27eb66bf1c01 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.478179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76859a4-3736-4c65-845c-27eb66bf1c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.478459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76859a4-3736-4c65-845c-27eb66bf1c01 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.479121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76859a4-3736-4c65-845c-27eb66bf1c01 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:56.479478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76859a4-3736-4c65-845c-27eb66bf1c01 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:56.481400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76859a4-3736-4c65-845c-27eb66bf1c01 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:56.489453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 62/117] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:40:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:56.494123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76859a4-3736-4c65-845c-27eb66bf1c01 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:56.516703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:56.527762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 56/118] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:56.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef22ee9-780c-49a7-8967-5bffd520ae01 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:56.610623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76859a4-3736-4c65-845c-27eb66bf1c01 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:56.657186 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef22ee9-780c-49a7-8967-5bffd520ae01 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:56.660274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:56.662978 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8fea86-3785-483d-a2d2-ca47b1d099e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:56.664122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8fea86-3785-483d-a2d2-ca47b1d099e2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.664604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8fea86-3785-483d-a2d2-ca47b1d099e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.665063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8fea86-3785-483d-a2d2-ca47b1d099e2 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.667743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8fea86-3785-483d-a2d2-ca47b1d099e2 None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:56.670084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8fea86-3785-483d-a2d2-ca47b1d099e2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:56.672707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 63/119] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:40:56 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:56.675516 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365535e3-62ed-418f-b85e-f345b5a46d17 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.675925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365535e3-62ed-418f-b85e-f345b5a46d17 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.676321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365535e3-62ed-418f-b85e-f345b5a46d17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.696668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-365535e3-62ed-418f-b85e-f345b5a46d17 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:56.698108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-365535e3-62ed-418f-b85e-f345b5a46d17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:56.707574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1effe023-9883-4d87-a832-06f5b3114cd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:56.708332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1effe023-9883-4d87-a832-06f5b3114cd0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.709130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1effe023-9883-4d87-a832-06f5b3114cd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.709538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1effe023-9883-4d87-a832-06f5b3114cd0 None admin] PATH_INFO: `/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.711187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1effe023-9883-4d87-a832-06f5b3114cd0 None admin] RBAC: Authorizing `identity:get_user(user_id=b3c5f8b9f24f40b99d0e28c5d77526fa)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:56.713004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1effe023-9883-4d87-a832-06f5b3114cd0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:56.715707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 57/120] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:56 2022] GET /identity/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa => generated 247 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:56.718902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1effe023-9883-4d87-a832-06f5b3114cd0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:56.776100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 64/121] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:56.777330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365535e3-62ed-418f-b85e-f345b5a46d17 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:56.798469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:56.850801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1effe023-9883-4d87-a832-06f5b3114cd0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:56.892885 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b191fd-23bb-4117-94a3-20f6bed1f700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:56.893673 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b191fd-23bb-4117-94a3-20f6bed1f700 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.894023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b191fd-23bb-4117-94a3-20f6bed1f700 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.894391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b191fd-23bb-4117-94a3-20f6bed1f700 None admin] PATH_INFO: `/v3/users/d7ae9bf858ce45f1887ed9fbf2036581` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.898658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b191fd-23bb-4117-94a3-20f6bed1f700 None admin] RBAC: Authorizing `identity:get_user(user_id=d7ae9bf858ce45f1887ed9fbf2036581)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:56.900305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b191fd-23bb-4117-94a3-20f6bed1f700 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:56.902085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 58/122] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:56 2022] GET /identity/v3/users/d7ae9bf858ce45f1887ed9fbf2036581 => generated 275 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:56.906269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b191fd-23bb-4117-94a3-20f6bed1f700 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:56.925332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365535e3-62ed-418f-b85e-f345b5a46d17 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:56.980193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d3d33b-f014-42a6-9828-7de6509af870 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:56.982086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3d33b-f014-42a6-9828-7de6509af870 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:56.982597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3d33b-f014-42a6-9828-7de6509af870 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:56.983710 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:56.984531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d3d33b-f014-42a6-9828-7de6509af870 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:56.985519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d3d33b-f014-42a6-9828-7de6509af870 None admin] RBAC: Adding query filter params (role.id=680bd58399004020984edb53af05d3f8, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e, user.id=d7c226f9fa6c4d79867811e016a8115e) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:56.986372 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d3d33b-f014-42a6-9828-7de6509af870 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:56.990457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d3d33b-f014-42a6-9828-7de6509af870 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:57.000506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 65/123] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:40:56 2022] GET /identity/v3/role_assignments?user.id=d7c226f9fa6c4d79867811e016a8115e&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=680bd58399004020984edb53af05d3f8 => generated 259 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:57.001666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d3d33b-f014-42a6-9828-7de6509af870 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:57.069121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b191fd-23bb-4117-94a3-20f6bed1f700 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:57.108344 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23388d57-616e-4919-9723-8a9fc9bd3374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:57.109501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23388d57-616e-4919-9723-8a9fc9bd3374 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:57.109501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23388d57-616e-4919-9723-8a9fc9bd3374 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:57.109867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23388d57-616e-4919-9723-8a9fc9bd3374 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:57.111233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23388d57-616e-4919-9723-8a9fc9bd3374 None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:57.112835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23388d57-616e-4919-9723-8a9fc9bd3374 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:57.114824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 59/124] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:40:56 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:57.115725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23388d57-616e-4919-9723-8a9fc9bd3374 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:57.139819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:57.194168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d3d33b-f014-42a6-9828-7de6509af870 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:57.267289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23388d57-616e-4919-9723-8a9fc9bd3374 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:57.277602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934c3fde-1566-4d1d-82eb-15acb5f21c41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:57.278653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934c3fde-1566-4d1d-82eb-15acb5f21c41 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:57.279133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934c3fde-1566-4d1d-82eb-15acb5f21c41 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:57.279637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934c3fde-1566-4d1d-82eb-15acb5f21c41 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:57.290085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:57.317662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934c3fde-1566-4d1d-82eb-15acb5f21c41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:57.319890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee25a805-4f27-48c7-a63c-cb3ada6a16b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:57.320526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee25a805-4f27-48c7-a63c-cb3ada6a16b0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:57.321292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee25a805-4f27-48c7-a63c-cb3ada6a16b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:57.322125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee25a805-4f27-48c7-a63c-cb3ada6a16b0 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:57.324812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee25a805-4f27-48c7-a63c-cb3ada6a16b0 None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:57.326291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee25a805-4f27-48c7-a63c-cb3ada6a16b0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-934c3fde-1566-4d1d-82eb-15acb5f21c41 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:57.326464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:57.327481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:40:57.328267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:40:57.328844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 60/125] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:40:57 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:57.329124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee25a805-4f27-48c7-a63c-cb3ada6a16b0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:57.332392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 66/126] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:40:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 330 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:57.332392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934c3fde-1566-4d1d-82eb-15acb5f21c41 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:57.488842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee25a805-4f27-48c7-a63c-cb3ada6a16b0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:57.531090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934c3fde-1566-4d1d-82eb-15acb5f21c41 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:57.531579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe6e476-71cf-4ce0-867d-9a61f44699e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:57.532124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe6e476-71cf-4ce0-867d-9a61f44699e8 None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:57.532380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe6e476-71cf-4ce0-867d-9a61f44699e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:57.532659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe6e476-71cf-4ce0-867d-9a61f44699e8 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/users/b3c5f8b9f24f40b99d0e28c5d77526fa/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:57.534621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe6e476-71cf-4ce0-867d-9a61f44699e8 None admin] RBAC: Authorizing `identity:create_grant(project_id=85be2a1b4ed54b91bfefe25b0945913e, user_id=b3c5f8b9f24f40b99d0e28c5d77526fa, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:57.536064 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe6e476-71cf-4ce0-867d-9a61f44699e8 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:57.551231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 61/127] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:40:57 2022] PUT /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/users/b3c5f8b9f24f40b99d0e28c5d77526fa/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:40:57.552131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe6e476-71cf-4ce0-867d-9a61f44699e8 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:57.573626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd2d39-c3d5-4bda-aaa8-00d1dc78af5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:57.574905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd2d39-c3d5-4bda-aaa8-00d1dc78af5e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:57.575408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd2d39-c3d5-4bda-aaa8-00d1dc78af5e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:57.575894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd2d39-c3d5-4bda-aaa8-00d1dc78af5e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:57.589221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffd2d39-c3d5-4bda-aaa8-00d1dc78af5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:40:57.591749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5ffd2d39-c3d5-4bda-aaa8-00d1dc78af5e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:40:57.597314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:40:57.598295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:40:57.599095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:40:57.606635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 67/128] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:40:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:40:57.608184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd2d39-c3d5-4bda-aaa8-00d1dc78af5e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:57.612887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:57.670906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe6e476-71cf-4ce0-867d-9a61f44699e8 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:57.708061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873e3eb-9090-4c0c-b35d-fa480b9cd476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:57.708743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873e3eb-9090-4c0c-b35d-fa480b9cd476 None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:57.709030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873e3eb-9090-4c0c-b35d-fa480b9cd476 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:57.709311 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e873e3eb-9090-4c0c-b35d-fa480b9cd476 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/users/d7ae9bf858ce45f1887ed9fbf2036581/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:57.711299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e873e3eb-9090-4c0c-b35d-fa480b9cd476 None admin] RBAC: Authorizing `identity:create_grant(project_id=85be2a1b4ed54b91bfefe25b0945913e, user_id=d7ae9bf858ce45f1887ed9fbf2036581, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:57.712777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e873e3eb-9090-4c0c-b35d-fa480b9cd476 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:57.732713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd2d39-c3d5-4bda-aaa8-00d1dc78af5e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:57.745068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 62/129] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:40:57 2022] PUT /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/users/d7ae9bf858ce45f1887ed9fbf2036581/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:40:57.745829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873e3eb-9090-4c0c-b35d-fa480b9cd476 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:57.777038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5020dfae-0911-42ee-a627-2553d27fe151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:57.778241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5020dfae-0911-42ee-a627-2553d27fe151 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:57.778769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5020dfae-0911-42ee-a627-2553d27fe151 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:57.779257 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5020dfae-0911-42ee-a627-2553d27fe151 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:57.780185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5020dfae-0911-42ee-a627-2553d27fe151 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:57.780688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5020dfae-0911-42ee-a627-2553d27fe151 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:57.782448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5020dfae-0911-42ee-a627-2553d27fe151 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:57.792277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 68/130] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:40:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:57.801938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5020dfae-0911-42ee-a627-2553d27fe151 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:57.874322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:57.901196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:57.908406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e873e3eb-9090-4c0c-b35d-fa480b9cd476 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:57.921849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5020dfae-0911-42ee-a627-2553d27fe151 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:57.948809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af598734-206e-40a7-bd88-9f2ae82403aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:57.949920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af598734-206e-40a7-bd88-9f2ae82403aa None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:57.950253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af598734-206e-40a7-bd88-9f2ae82403aa None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:57.950531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af598734-206e-40a7-bd88-9f2ae82403aa None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:57.951341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af598734-206e-40a7-bd88-9f2ae82403aa None admin] RBAC: Adding query filter params (name=member) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:40:57.951586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af598734-206e-40a7-bd88-9f2ae82403aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:57.952971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af598734-206e-40a7-bd88-9f2ae82403aa None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:57.966084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 63/131] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:40:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:57.971077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb52175-f149-48b7-b2bb-0d36aaa366ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:57.971343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af598734-206e-40a7-bd88-9f2ae82403aa None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:57.972334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb52175-f149-48b7-b2bb-0d36aaa366ce None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:57.972828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb52175-f149-48b7-b2bb-0d36aaa366ce None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:57.973336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb52175-f149-48b7-b2bb-0d36aaa366ce None admin] PATH_INFO: `/v3/users/d7ae9bf858ce45f1887ed9fbf2036581` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:57.974810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb52175-f149-48b7-b2bb-0d36aaa366ce None admin] RBAC: Authorizing `identity:get_user(user_id=d7ae9bf858ce45f1887ed9fbf2036581)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:57.976501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb52175-f149-48b7-b2bb-0d36aaa366ce None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:57.978642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 69/132] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:57 2022] GET /identity/v3/users/d7ae9bf858ce45f1887ed9fbf2036581 => generated 275 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:57.984991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb52175-f149-48b7-b2bb-0d36aaa366ce None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:58.135690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af598734-206e-40a7-bd88-9f2ae82403aa None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:58.169607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb52175-f149-48b7-b2bb-0d36aaa366ce None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:58.174501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa145aa0-0e69-4c46-b0f6-52795c111815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:58.175146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa145aa0-0e69-4c46-b0f6-52795c111815 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:58.175430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa145aa0-0e69-4c46-b0f6-52795c111815 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:58.175708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa145aa0-0e69-4c46-b0f6-52795c111815 None admin] PATH_INFO: `/v3/users/d7ae9bf858ce45f1887ed9fbf2036581` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:58.177237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa145aa0-0e69-4c46-b0f6-52795c111815 None admin] RBAC: Authorizing `identity:get_user(user_id=d7ae9bf858ce45f1887ed9fbf2036581)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:58.178757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa145aa0-0e69-4c46-b0f6-52795c111815 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:58.182883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 64/133] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:40:57 2022] GET /identity/v3/users/d7ae9bf858ce45f1887ed9fbf2036581 => generated 275 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:58.187942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa145aa0-0e69-4c46-b0f6-52795c111815 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:58.211433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:58.223359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:58.224990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4423f9d0-ea07-439f-88f2-aa471458b2ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:58.225851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4423f9d0-ea07-439f-88f2-aa471458b2ff None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:58.226370 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4423f9d0-ea07-439f-88f2-aa471458b2ff None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:58.226796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4423f9d0-ea07-439f-88f2-aa471458b2ff None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:58.228177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4423f9d0-ea07-439f-88f2-aa471458b2ff None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:58.229834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4423f9d0-ea07-439f-88f2-aa471458b2ff None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:58.231920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 70/134] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:40:57 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:58.238444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4423f9d0-ea07-439f-88f2-aa471458b2ff None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:58.302173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa145aa0-0e69-4c46-b0f6-52795c111815 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:58.335632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fd4934-1303-4d58-94fe-2589be4f6367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:58.336331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fd4934-1303-4d58-94fe-2589be4f6367 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:58.336563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fd4934-1303-4d58-94fe-2589be4f6367 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:58.336828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fd4934-1303-4d58-94fe-2589be4f6367 None admin] PATH_INFO: `/v3/projects/449b012a1fc24571bd474f23d9c1f706` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:58.337961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fd4934-1303-4d58-94fe-2589be4f6367 None admin] RBAC: Authorizing `identity:get_project(project_id=449b012a1fc24571bd474f23d9c1f706)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:58.339536 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fd4934-1303-4d58-94fe-2589be4f6367 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:58.346421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 65/135] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:40:58 2022] GET /identity/v3/projects/449b012a1fc24571bd474f23d9c1f706 => generated 312 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:40:58.354514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fd4934-1303-4d58-94fe-2589be4f6367 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:58.438541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4423f9d0-ea07-439f-88f2-aa471458b2ff None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:58.504314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c50bc58-5db2-411b-9e70-7fb6710645f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:58.504994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c50bc58-5db2-411b-9e70-7fb6710645f5 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:58.505275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c50bc58-5db2-411b-9e70-7fb6710645f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:58.505549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c50bc58-5db2-411b-9e70-7fb6710645f5 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/users/d7ae9bf858ce45f1887ed9fbf2036581/roles/5e19a16e5bfd4c94b1c34ae717ccfc42` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:58.512817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c50bc58-5db2-411b-9e70-7fb6710645f5 None admin] RBAC: Authorizing `identity:create_grant(project_id=85be2a1b4ed54b91bfefe25b0945913e, user_id=d7ae9bf858ce45f1887ed9fbf2036581, role_id=5e19a16e5bfd4c94b1c34ae717ccfc42)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:58.514406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c50bc58-5db2-411b-9e70-7fb6710645f5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:58.536239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 71/136] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:40:58 2022] PUT /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/users/d7ae9bf858ce45f1887ed9fbf2036581/roles/5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:40:58.537673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c152466c-34fb-4872-a24d-6cb62ddf7eb9 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:58.537967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c152466c-34fb-4872-a24d-6cb62ddf7eb9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:58.540059 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c152466c-34fb-4872-a24d-6cb62ddf7eb9 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:58.545086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 72/137] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:58 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:58.551827 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2158cd-48db-4afb-aced-e1a54024362c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:58.552150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2158cd-48db-4afb-aced-e1a54024362c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:58.552434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2158cd-48db-4afb-aced-e1a54024362c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:58.560178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:58.576919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fd4934-1303-4d58-94fe-2589be4f6367 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:58.591166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bf2158cd-48db-4afb-aced-e1a54024362c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:58.592393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bf2158cd-48db-4afb-aced-e1a54024362c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:58.644666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a14672f-6d0b-49d9-8710-523be8e13688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:58.645526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a14672f-6d0b-49d9-8710-523be8e13688 None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:58.645903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a14672f-6d0b-49d9-8710-523be8e13688 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:58.646238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a14672f-6d0b-49d9-8710-523be8e13688 None admin] PATH_INFO: `/v3/projects/449b012a1fc24571bd474f23d9c1f706/users/d7ae9bf858ce45f1887ed9fbf2036581/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:58.652536 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a14672f-6d0b-49d9-8710-523be8e13688 None admin] RBAC: Authorizing `identity:create_grant(project_id=449b012a1fc24571bd474f23d9c1f706, user_id=d7ae9bf858ce45f1887ed9fbf2036581, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:58.653336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a14672f-6d0b-49d9-8710-523be8e13688 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:58.684968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 66/138] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:40:58 2022] PUT /identity/v3/projects/449b012a1fc24571bd474f23d9c1f706/users/d7ae9bf858ce45f1887ed9fbf2036581/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:40:58.702718 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:40:58.728963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 73/139] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:40:59.055607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a14672f-6d0b-49d9-8710-523be8e13688 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:59.179278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a14672f-6d0b-49d9-8710-523be8e13688 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:59.231306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763653b1-a7cf-4e79-b814-65590a186723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:59.231945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763653b1-a7cf-4e79-b814-65590a186723 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:59.232221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763653b1-a7cf-4e79-b814-65590a186723 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:59.232493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763653b1-a7cf-4e79-b814-65590a186723 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:59.233643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763653b1-a7cf-4e79-b814-65590a186723 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:59.235109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763653b1-a7cf-4e79-b814-65590a186723 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:59.236835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 67/140] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:40:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:40:59.242136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2158cd-48db-4afb-aced-e1a54024362c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:40:59.364834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2158cd-48db-4afb-aced-e1a54024362c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:40:59.406264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ecf8b4-5e46-4245-9c7c-c8749c58d803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:40:59.406871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ecf8b4-5e46-4245-9c7c-c8749c58d803 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:59.407144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ecf8b4-5e46-4245-9c7c-c8749c58d803 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:59.407414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ecf8b4-5e46-4245-9c7c-c8749c58d803 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:59.408067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ecf8b4-5e46-4245-9c7c-c8749c58d803 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:40:59.409577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ecf8b4-5e46-4245-9c7c-c8749c58d803 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:40:59.436373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 74/141] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:40:59 2022] POST /identity/v3/users => generated 283 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:40:59.891236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6007a58-8196-4fc4-9dba-bf6c34dbf947 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:59.891739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6007a58-8196-4fc4-9dba-bf6c34dbf947 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:59.892019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6007a58-8196-4fc4-9dba-bf6c34dbf947 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:59.893210 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 68/142] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:40:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:40:59.907041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef89be03-5524-4482-b810-aec1ae04f239 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:40:59.907506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef89be03-5524-4482-b810-aec1ae04f239 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:40:59.907577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef89be03-5524-4482-b810-aec1ae04f239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:40:59.930531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ef89be03-5524-4482-b810-aec1ae04f239 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:40:59.931743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ef89be03-5524-4482-b810-aec1ae04f239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:40:59.989653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:00.021068 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 75/143] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:40:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:00.590220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6007a58-8196-4fc4-9dba-bf6c34dbf947 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:00.616126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:00.698004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344f8a50-0b77-4777-bdf4-e1cea117b7bb None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:00.698309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344f8a50-0b77-4777-bdf4-e1cea117b7bb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:00.698727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344f8a50-0b77-4777-bdf4-e1cea117b7bb None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:00.700190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 76/144] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:41:00.705496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63041db2-b854-45a1-a92a-71b5c1cf8aad None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:00.705778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63041db2-b854-45a1-a92a-71b5c1cf8aad None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:00.706081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63041db2-b854-45a1-a92a-71b5c1cf8aad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:00.709121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6007a58-8196-4fc4-9dba-bf6c34dbf947 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:00.724786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-63041db2-b854-45a1-a92a-71b5c1cf8aad None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:00.725966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-63041db2-b854-45a1-a92a-71b5c1cf8aad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:00.748444 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd72508-66c4-4158-ae0b-29b4db742fd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:00.749882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd72508-66c4-4158-ae0b-29b4db742fd7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:00.750404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd72508-66c4-4158-ae0b-29b4db742fd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:00.751026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd72508-66c4-4158-ae0b-29b4db742fd7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:00.763023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd72508-66c4-4158-ae0b-29b4db742fd7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-cdd72508-66c4-4158-ae0b-29b4db742fd7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:00.769870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:00.770811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:00.771626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:00.774018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 69/145] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:00 2022] GET /identity/v3/roles/reader => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:00.779830 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd72508-66c4-4158-ae0b-29b4db742fd7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:00.810315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 77/146] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:00.877136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:00.891027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd72508-66c4-4158-ae0b-29b4db742fd7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:00.928983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003880aa-a7de-4054-9037-2309b8d19672 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:00.930451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003880aa-a7de-4054-9037-2309b8d19672 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:00.930894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003880aa-a7de-4054-9037-2309b8d19672 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:00.931332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003880aa-a7de-4054-9037-2309b8d19672 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:00.932162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003880aa-a7de-4054-9037-2309b8d19672 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:00.932628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003880aa-a7de-4054-9037-2309b8d19672 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:00.934276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003880aa-a7de-4054-9037-2309b8d19672 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:00.943713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 70/147] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:00.955185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63041db2-b854-45a1-a92a-71b5c1cf8aad None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:01.089412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63041db2-b854-45a1-a92a-71b5c1cf8aad None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:01.103300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:01.128971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6389fa-8384-4fab-bc1e-871e0d3498f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:01.129521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6389fa-8384-4fab-bc1e-871e0d3498f6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:01.129797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6389fa-8384-4fab-bc1e-871e0d3498f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:01.130089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6389fa-8384-4fab-bc1e-871e0d3498f6 None admin] PATH_INFO: `/v3/users/d7c226f9fa6c4d79867811e016a8115e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:01.131193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6389fa-8384-4fab-bc1e-871e0d3498f6 None admin] RBAC: Authorizing `identity:get_user(user_id=d7c226f9fa6c4d79867811e016a8115e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:01.132632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6389fa-8384-4fab-bc1e-871e0d3498f6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:01.134404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 78/148] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:00 2022] GET /identity/v3/users/d7c226f9fa6c4d79867811e016a8115e => generated 289 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:01.140278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003880aa-a7de-4054-9037-2309b8d19672 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:01.202432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dcbf29-346b-4a54-b2ec-e03abe93b737 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:01.202748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dcbf29-346b-4a54-b2ec-e03abe93b737 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:01.202972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dcbf29-346b-4a54-b2ec-e03abe93b737 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:01.204509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 79/149] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:41:01.209580 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22bdd1b-6630-43f4-8430-f854a6250773 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:01.209856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22bdd1b-6630-43f4-8430-f854a6250773 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:01.210153 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22bdd1b-6630-43f4-8430-f854a6250773 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:01.231325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e22bdd1b-6630-43f4-8430-f854a6250773 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:01.232959 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e22bdd1b-6630-43f4-8430-f854a6250773 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:01.273506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003880aa-a7de-4054-9037-2309b8d19672 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:01.319929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acabc743-503c-4b68-b23f-de06fdd60f76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:01.320685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 80/150] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:01.321416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acabc743-503c-4b68-b23f-de06fdd60f76 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:01.321974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acabc743-503c-4b68-b23f-de06fdd60f76 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:01.322496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acabc743-503c-4b68-b23f-de06fdd60f76 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:01.325142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acabc743-503c-4b68-b23f-de06fdd60f76 None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:01.326890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acabc743-503c-4b68-b23f-de06fdd60f76 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:01.328748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 71/151] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:01 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:01.330814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22bdd1b-6630-43f4-8430-f854a6250773 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:01.357030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acabc743-503c-4b68-b23f-de06fdd60f76 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:01.386054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:01.450198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22bdd1b-6630-43f4-8430-f854a6250773 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:01.458500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:01.485679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740f383-7ca5-4949-8a98-4eaf375e5a3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:01.486356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740f383-7ca5-4949-8a98-4eaf375e5a3e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:01.486626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740f383-7ca5-4949-8a98-4eaf375e5a3e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:01.486893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740f383-7ca5-4949-8a98-4eaf375e5a3e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:01.506891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b740f383-7ca5-4949-8a98-4eaf375e5a3e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b740f383-7ca5-4949-8a98-4eaf375e5a3e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:01.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:01.516932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:01.517764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 81/152] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:41:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:01.522928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740f383-7ca5-4949-8a98-4eaf375e5a3e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:01.537972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acabc743-503c-4b68-b23f-de06fdd60f76 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:01.627756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff037de2-211e-49e3-8cb0-804ab25b1fd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:01.628781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff037de2-211e-49e3-8cb0-804ab25b1fd8 None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:01.629226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff037de2-211e-49e3-8cb0-804ab25b1fd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:01.629653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff037de2-211e-49e3-8cb0-804ab25b1fd8 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/users/d7c226f9fa6c4d79867811e016a8115e/roles/680bd58399004020984edb53af05d3f8` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:01.635163 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff037de2-211e-49e3-8cb0-804ab25b1fd8 None admin] RBAC: Authorizing `identity:create_grant(project_id=85be2a1b4ed54b91bfefe25b0945913e, user_id=d7c226f9fa6c4d79867811e016a8115e, role_id=680bd58399004020984edb53af05d3f8)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:01.638279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff037de2-211e-49e3-8cb0-804ab25b1fd8 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:01.664737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740f383-7ca5-4949-8a98-4eaf375e5a3e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:01.667355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 72/153] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:41:01 2022] PUT /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/users/d7c226f9fa6c4d79867811e016a8115e/roles/680bd58399004020984edb53af05d3f8 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:41:01.727839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccc5e5b-71a4-4efb-923f-34faf062a5fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:01.728609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccc5e5b-71a4-4efb-923f-34faf062a5fb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:01.728882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccc5e5b-71a4-4efb-923f-34faf062a5fb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:01.729148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccc5e5b-71a4-4efb-923f-34faf062a5fb None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:01.729816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccc5e5b-71a4-4efb-923f-34faf062a5fb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:01.730706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccc5e5b-71a4-4efb-923f-34faf062a5fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:01.732121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccc5e5b-71a4-4efb-923f-34faf062a5fb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:01.741804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 82/154] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:01.747224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff037de2-211e-49e3-8cb0-804ab25b1fd8 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:01.847035 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:01.887081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff037de2-211e-49e3-8cb0-804ab25b1fd8 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:01.930575 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaff4e9-2b74-4756-8e6e-8f181d0ca8c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:01.931169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaff4e9-2b74-4756-8e6e-8f181d0ca8c6 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:01.931419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaff4e9-2b74-4756-8e6e-8f181d0ca8c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:01.931697 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaff4e9-2b74-4756-8e6e-8f181d0ca8c6 None admin] PATH_INFO: `/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:01.932945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaff4e9-2b74-4756-8e6e-8f181d0ca8c6 None admin] RBAC: Authorizing `identity:get_user(user_id=b3c5f8b9f24f40b99d0e28c5d77526fa)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:01.934999 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaff4e9-2b74-4756-8e6e-8f181d0ca8c6 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:01.936600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 73/155] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:01 2022] GET /identity/v3/users/b3c5f8b9f24f40b99d0e28c5d77526fa => generated 247 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:01.942327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccc5e5b-71a4-4efb-923f-34faf062a5fb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:01.984790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:02.078679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccc5e5b-71a4-4efb-923f-34faf062a5fb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:02.122246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04490c2-cb5a-4d76-b999-a43054dd1aec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:02.123127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04490c2-cb5a-4d76-b999-a43054dd1aec None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:02.123549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04490c2-cb5a-4d76-b999-a43054dd1aec None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:02.123949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04490c2-cb5a-4d76-b999-a43054dd1aec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:02.153410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaff4e9-2b74-4756-8e6e-8f181d0ca8c6 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:02.209471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04490c2-cb5a-4d76-b999-a43054dd1aec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:02.211047 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04490c2-cb5a-4d76-b999-a43054dd1aec None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:02.246428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:02.345587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaff4e9-2b74-4756-8e6e-8f181d0ca8c6 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:02.346618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 83/156] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:02.352448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04490c2-cb5a-4d76-b999-a43054dd1aec None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:02.451145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f9e0f9-2dcb-423c-bda2-dfe10f114cab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:02.452064 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f9e0f9-2dcb-423c-bda2-dfe10f114cab None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:02.452483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f9e0f9-2dcb-423c-bda2-dfe10f114cab None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:02.452905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f9e0f9-2dcb-423c-bda2-dfe10f114cab None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:02.472921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:02.482185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f9e0f9-2dcb-423c-bda2-dfe10f114cab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-04f9e0f9-2dcb-423c-bda2-dfe10f114cab None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:02.490856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:02.491860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:02.492643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 74/157] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 339 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:02.497978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f9e0f9-2dcb-423c-bda2-dfe10f114cab None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:02.499517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04490c2-cb5a-4d76-b999-a43054dd1aec None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:02.533732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:02.552181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a05f05b-2d5d-4da8-baed-4511df74cddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:02.552871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a05f05b-2d5d-4da8-baed-4511df74cddf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:02.553143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a05f05b-2d5d-4da8-baed-4511df74cddf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:02.553517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a05f05b-2d5d-4da8-baed-4511df74cddf None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:02.556510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a05f05b-2d5d-4da8-baed-4511df74cddf None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:02.558011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a05f05b-2d5d-4da8-baed-4511df74cddf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:02.560195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 84/158] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:02 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:02.565608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a05f05b-2d5d-4da8-baed-4511df74cddf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:02.684772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f9e0f9-2dcb-423c-bda2-dfe10f114cab None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:02.750171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a05f05b-2d5d-4da8-baed-4511df74cddf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:02.756209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c365988-8649-457e-b039-adaff220ed23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:02.756822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c365988-8649-457e-b039-adaff220ed23 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:02.757108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c365988-8649-457e-b039-adaff220ed23 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:02.757395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c365988-8649-457e-b039-adaff220ed23 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:02.758085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c365988-8649-457e-b039-adaff220ed23 None admin] RBAC: Adding query filter params (name=member) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:02.758398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c365988-8649-457e-b039-adaff220ed23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:02.759782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c365988-8649-457e-b039-adaff220ed23 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:02.774087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 75/159] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:02.775800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4cc549-5216-40d5-98e4-0daf77a88229 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:02.776093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4cc549-5216-40d5-98e4-0daf77a88229 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:02.776372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4cc549-5216-40d5-98e4-0daf77a88229 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:02.778252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 76/160] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:02.778962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4cc549-5216-40d5-98e4-0daf77a88229 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:02.796139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24084e7-d0ac-4887-bf88-27cae6f32ec6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:02.797146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24084e7-d0ac-4887-bf88-27cae6f32ec6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:02.797649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24084e7-d0ac-4887-bf88-27cae6f32ec6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:02.798211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24084e7-d0ac-4887-bf88-27cae6f32ec6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:02.799213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24084e7-d0ac-4887-bf88-27cae6f32ec6 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e, user.id=b3c5f8b9f24f40b99d0e28c5d77526fa) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:02.799738 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24084e7-d0ac-4887-bf88-27cae6f32ec6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:02.801490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24084e7-d0ac-4887-bf88-27cae6f32ec6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:02.809559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 85/161] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:02 2022] GET /identity/v3/role_assignments?user.id=b3c5f8b9f24f40b99d0e28c5d77526fa&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 608 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:02.811418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b27f6b-cf9e-4398-add2-0a31e612f229 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:02.811922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b27f6b-cf9e-4398-add2-0a31e612f229 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:02.812407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b27f6b-cf9e-4398-add2-0a31e612f229 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:02.833569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f8b27f6b-cf9e-4398-add2-0a31e612f229 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:02.835238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f8b27f6b-cf9e-4398-add2-0a31e612f229 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:02.899755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:02.945453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:02.978200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 86/162] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:02.979520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc5a30-d963-43f8-8a09-f0629a0cc747 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:02.979804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc5a30-d963-43f8-8a09-f0629a0cc747 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:02.981121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc5a30-d963-43f8-8a09-f0629a0cc747 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:02.982498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 87/163] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:02.987361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4cc549-5216-40d5-98e4-0daf77a88229 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:02.993247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd0bfd2-c063-476f-b78a-ead5406337d9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:02.995671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd0bfd2-c063-476f-b78a-ead5406337d9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:02.996031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd0bfd2-c063-476f-b78a-ead5406337d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:03.049117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2bd0bfd2-c063-476f-b78a-ead5406337d9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:03.050419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2bd0bfd2-c063-476f-b78a-ead5406337d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:03.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14199971-6735-4832-86b8-92f26e1a8c7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:03.081275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14199971-6735-4832-86b8-92f26e1a8c7a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:03.081619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14199971-6735-4832-86b8-92f26e1a8c7a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:03.081899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14199971-6735-4832-86b8-92f26e1a8c7a None admin] PATH_INFO: `/v3/users/d7ae9bf858ce45f1887ed9fbf2036581` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:03.083744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14199971-6735-4832-86b8-92f26e1a8c7a None admin] RBAC: Authorizing `identity:get_user(user_id=d7ae9bf858ce45f1887ed9fbf2036581)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:03.086295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14199971-6735-4832-86b8-92f26e1a8c7a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:03.088957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 77/164] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:02 2022] GET /identity/v3/users/d7ae9bf858ce45f1887ed9fbf2036581 => generated 275 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:03.095394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14199971-6735-4832-86b8-92f26e1a8c7a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:03.228298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 88/165] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:03.243852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14199971-6735-4832-86b8-92f26e1a8c7a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:03.274623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:03.286153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fb1775-1baf-420e-8f6d-c1e6792365f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:03.286707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fb1775-1baf-420e-8f6d-c1e6792365f0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:03.286988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fb1775-1baf-420e-8f6d-c1e6792365f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:03.287265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fb1775-1baf-420e-8f6d-c1e6792365f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:03.362750 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fb1775-1baf-420e-8f6d-c1e6792365f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:03.364281 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fb1775-1baf-420e-8f6d-c1e6792365f0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:03.486208 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 78/166] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:03.491885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd0bfd2-c063-476f-b78a-ead5406337d9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:03.557346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fb1775-1baf-420e-8f6d-c1e6792365f0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:03.612365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:03.659622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd0bfd2-c063-476f-b78a-ead5406337d9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:03.693579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:03.724104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491ba417-7a35-4c6b-9825-1e8820186b02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:03.726114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491ba417-7a35-4c6b-9825-1e8820186b02 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:03.727202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491ba417-7a35-4c6b-9825-1e8820186b02 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:03.727829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491ba417-7a35-4c6b-9825-1e8820186b02 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:03.730694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491ba417-7a35-4c6b-9825-1e8820186b02 None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:03.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491ba417-7a35-4c6b-9825-1e8820186b02 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:03.735180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 89/167] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:03 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:03.741101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fb1775-1baf-420e-8f6d-c1e6792365f0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:03.742199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491ba417-7a35-4c6b-9825-1e8820186b02 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:03.798729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af724069-6694-4987-9f3e-a574553386bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:03.799716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af724069-6694-4987-9f3e-a574553386bf None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:03.800304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af724069-6694-4987-9f3e-a574553386bf None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:03.800771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af724069-6694-4987-9f3e-a574553386bf None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:03.814398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af724069-6694-4987-9f3e-a574553386bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-af724069-6694-4987-9f3e-a574553386bf None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:03.820026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:03.821048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:03.821828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:03.830452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 79/168] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:03.831563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af724069-6694-4987-9f3e-a574553386bf None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:03.928562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:03.941217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af724069-6694-4987-9f3e-a574553386bf None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:03.966291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491ba417-7a35-4c6b-9825-1e8820186b02 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:04.003008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0373585-7d21-45d1-b863-25a4d59c62d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:04.006881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0373585-7d21-45d1-b863-25a4d59c62d0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:04.007368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0373585-7d21-45d1-b863-25a4d59c62d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:04.007887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0373585-7d21-45d1-b863-25a4d59c62d0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:04.033043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da23f0b2-a04b-4817-b78a-bb99cfcfec13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:04.034371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da23f0b2-a04b-4817-b78a-bb99cfcfec13 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:04.034849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da23f0b2-a04b-4817-b78a-bb99cfcfec13 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:04.035336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da23f0b2-a04b-4817-b78a-bb99cfcfec13 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:04.037122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da23f0b2-a04b-4817-b78a-bb99cfcfec13 None admin] RBAC: Adding query filter params (role.id=e4fa1b2d8c404336a855b6142fbf25d4, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e, user.id=d7ae9bf858ce45f1887ed9fbf2036581) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:04.037630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da23f0b2-a04b-4817-b78a-bb99cfcfec13 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:04.038529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0373585-7d21-45d1-b863-25a4d59c62d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-f0373585-7d21-45d1-b863-25a4d59c62d0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:04.044494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:04.045369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:04.046213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:04.050144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da23f0b2-a04b-4817-b78a-bb99cfcfec13 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:04.050624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 80/169] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:04.051620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0373585-7d21-45d1-b863-25a4d59c62d0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:04.061805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 90/170] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:03 2022] GET /identity/v3/role_assignments?user.id=d7ae9bf858ce45f1887ed9fbf2036581&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 608 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:04.063016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da23f0b2-a04b-4817-b78a-bb99cfcfec13 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:04.144238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:04.210396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0373585-7d21-45d1-b863-25a4d59c62d0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:04.214095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da23f0b2-a04b-4817-b78a-bb99cfcfec13 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:04.284778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf37992-d302-4ec1-8cc2-b1dc071c9903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:04.285147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59cc6cc-fd2a-4bde-9b2f-569c0c2eaf54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:04.286005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59cc6cc-fd2a-4bde-9b2f-569c0c2eaf54 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:04.286825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf37992-d302-4ec1-8cc2-b1dc071c9903 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:04.287296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59cc6cc-fd2a-4bde-9b2f-569c0c2eaf54 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:04.287775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59cc6cc-fd2a-4bde-9b2f-569c0c2eaf54 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:04.288255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf37992-d302-4ec1-8cc2-b1dc071c9903 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:04.289123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59cc6cc-fd2a-4bde-9b2f-569c0c2eaf54 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:04.289610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf37992-d302-4ec1-8cc2-b1dc071c9903 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:04.290502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf37992-d302-4ec1-8cc2-b1dc071c9903 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:04.291022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59cc6cc-fd2a-4bde-9b2f-569c0c2eaf54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:04.292676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59cc6cc-fd2a-4bde-9b2f-569c0c2eaf54 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:04.293207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf37992-d302-4ec1-8cc2-b1dc071c9903 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:04.294919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf37992-d302-4ec1-8cc2-b1dc071c9903 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:04.313033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 91/171] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:04.315827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 81/172] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:04.319133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59cc6cc-fd2a-4bde-9b2f-569c0c2eaf54 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:04.322146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf37992-d302-4ec1-8cc2-b1dc071c9903 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:04.347773 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:04.490318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59cc6cc-fd2a-4bde-9b2f-569c0c2eaf54 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:04.501462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf37992-d302-4ec1-8cc2-b1dc071c9903 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:04.518453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:04.554018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1225f8f-e0c0-4535-9372-43d501370632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:04.554722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1225f8f-e0c0-4535-9372-43d501370632 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:04.554974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1225f8f-e0c0-4535-9372-43d501370632 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:04.555249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1225f8f-e0c0-4535-9372-43d501370632 None admin] PATH_INFO: `/v3/users/d7ae9bf858ce45f1887ed9fbf2036581` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:04.556841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1225f8f-e0c0-4535-9372-43d501370632 None admin] RBAC: Authorizing `identity:get_user(user_id=d7ae9bf858ce45f1887ed9fbf2036581)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:04.558244 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1225f8f-e0c0-4535-9372-43d501370632 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:04.560008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 82/173] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:04 2022] GET /identity/v3/users/d7ae9bf858ce45f1887ed9fbf2036581 => generated 275 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:04.565251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1225f8f-e0c0-4535-9372-43d501370632 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:04.586835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded6b562-3d2a-4a57-b434-a92aeda3011a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:04.587480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded6b562-3d2a-4a57-b434-a92aeda3011a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:04.587800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded6b562-3d2a-4a57-b434-a92aeda3011a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:04.588074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded6b562-3d2a-4a57-b434-a92aeda3011a None admin] PATH_INFO: `/v3/users/d7ae9bf858ce45f1887ed9fbf2036581` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:04.589279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded6b562-3d2a-4a57-b434-a92aeda3011a None admin] RBAC: Authorizing `identity:get_user(user_id=d7ae9bf858ce45f1887ed9fbf2036581)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:04.590780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded6b562-3d2a-4a57-b434-a92aeda3011a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:04.592340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 92/174] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:04 2022] GET /identity/v3/users/d7ae9bf858ce45f1887ed9fbf2036581 => generated 275 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:04.615725 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded6b562-3d2a-4a57-b434-a92aeda3011a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:04.659459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:04.682198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1225f8f-e0c0-4535-9372-43d501370632 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:04.725496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f46bc0-528d-46b7-ae81-d761c5d5275c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:04.726148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f46bc0-528d-46b7-ae81-d761c5d5275c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:04.726430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f46bc0-528d-46b7-ae81-d761c5d5275c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:04.726703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f46bc0-528d-46b7-ae81-d761c5d5275c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:04.850809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded6b562-3d2a-4a57-b434-a92aeda3011a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:04.868060 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f46bc0-528d-46b7-ae81-d761c5d5275c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:04.869532 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f46bc0-528d-46b7-ae81-d761c5d5275c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:04.916452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d45abc-0830-4cf1-bc06-fc423d6fa14f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:04.917066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d45abc-0830-4cf1-bc06-fc423d6fa14f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:04.917345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d45abc-0830-4cf1-bc06-fc423d6fa14f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:04.917724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d45abc-0830-4cf1-bc06-fc423d6fa14f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:04.976704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:04.982349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:05.002418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d45abc-0830-4cf1-bc06-fc423d6fa14f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:05.004117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d45abc-0830-4cf1-bc06-fc423d6fa14f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:05.008942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 83/175] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 02 00:41:05.011315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965432ff-f8ac-4fc1-aeb8-a43ce58cc6a9 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.011818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965432ff-f8ac-4fc1-aeb8-a43ce58cc6a9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.012291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965432ff-f8ac-4fc1-aeb8-a43ce58cc6a9 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.013838 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 84/176] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:05.019119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965432ff-f8ac-4fc1-aeb8-a43ce58cc6a9 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:05.131007 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 93/177] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 516 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 02 00:41:05.133195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c86f0b-fff2-4652-b76c-9f76ee6bc5aa None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.133695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c86f0b-fff2-4652-b76c-9f76ee6bc5aa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.134239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c86f0b-fff2-4652-b76c-9f76ee6bc5aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.153384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965432ff-f8ac-4fc1-aeb8-a43ce58cc6a9 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:05.157554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c0c86f0b-fff2-4652-b76c-9f76ee6bc5aa None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:05.159004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c0c86f0b-fff2-4652-b76c-9f76ee6bc5aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:05.194520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa9ee01-8435-44ec-bb21-221e96303e6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:05.195156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa9ee01-8435-44ec-bb21-221e96303e6b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.195522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa9ee01-8435-44ec-bb21-221e96303e6b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.195810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa9ee01-8435-44ec-bb21-221e96303e6b None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.197032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa9ee01-8435-44ec-bb21-221e96303e6b None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:05.198847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa9ee01-8435-44ec-bb21-221e96303e6b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:05.201508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 85/178] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:05 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:05.202337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa9ee01-8435-44ec-bb21-221e96303e6b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:05.241398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 94/179] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:05.246158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c86f0b-fff2-4652-b76c-9f76ee6bc5aa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:05.255104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:05.274976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:05.361821 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c86f0b-fff2-4652-b76c-9f76ee6bc5aa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:05.403773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8404b39-a815-4be7-b101-5b264f7a763e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:05.404284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa9ee01-8435-44ec-bb21-221e96303e6b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:05.404394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8404b39-a815-4be7-b101-5b264f7a763e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.404589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8404b39-a815-4be7-b101-5b264f7a763e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.404857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8404b39-a815-4be7-b101-5b264f7a763e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.405586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8404b39-a815-4be7-b101-5b264f7a763e None admin] RBAC: Adding query filter params (role.id=5e19a16e5bfd4c94b1c34ae717ccfc42, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e, user.id=d7ae9bf858ce45f1887ed9fbf2036581) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:05.405889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8404b39-a815-4be7-b101-5b264f7a763e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:05.407525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8404b39-a815-4be7-b101-5b264f7a763e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:05.414599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 95/180] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:05 2022] GET /identity/v3/role_assignments?user.id=d7ae9bf858ce45f1887ed9fbf2036581&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:05.415836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1809458-b3bb-41c7-a27d-8a8c6baf95eb None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.416117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1809458-b3bb-41c7-a27d-8a8c6baf95eb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.416383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1809458-b3bb-41c7-a27d-8a8c6baf95eb None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.418128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 96/181] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:05.424464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09236ab-11e8-4b49-b55a-c14cc970b915 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.424772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09236ab-11e8-4b49-b55a-c14cc970b915 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.425034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09236ab-11e8-4b49-b55a-c14cc970b915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.447859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b09236ab-11e8-4b49-b55a-c14cc970b915 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:05.450239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b09236ab-11e8-4b49-b55a-c14cc970b915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:05.494802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea91592-4472-458b-8dcc-350ab89914f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:05.495490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea91592-4472-458b-8dcc-350ab89914f9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.495841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea91592-4472-458b-8dcc-350ab89914f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.496111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea91592-4472-458b-8dcc-350ab89914f9 None admin] PATH_INFO: `/v3/projects/449b012a1fc24571bd474f23d9c1f706` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.501795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea91592-4472-458b-8dcc-350ab89914f9 None admin] RBAC: Authorizing `identity:get_project(project_id=449b012a1fc24571bd474f23d9c1f706)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:05.503307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea91592-4472-458b-8dcc-350ab89914f9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:05.505188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 86/182] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:05 2022] GET /identity/v3/projects/449b012a1fc24571bd474f23d9c1f706 => generated 312 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:05.510999 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea91592-4472-458b-8dcc-350ab89914f9 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:05.576309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 97/183] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:05.591361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6cbc89-2214-4822-8f06-3f917f44a75f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.591951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6cbc89-2214-4822-8f06-3f917f44a75f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.592434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6cbc89-2214-4822-8f06-3f917f44a75f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.594121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 98/184] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:05.595352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfea326-66a9-40c9-a1bd-3f2aad9d176e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.595901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfea326-66a9-40c9-a1bd-3f2aad9d176e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.596367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfea326-66a9-40c9-a1bd-3f2aad9d176e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.597781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 99/185] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:05.603548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e3a10b-2196-47c9-a594-9f6fe37ab4fd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.604174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e3a10b-2196-47c9-a594-9f6fe37ab4fd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.604660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e3a10b-2196-47c9-a594-9f6fe37ab4fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.615301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:05.628795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-05e3a10b-2196-47c9-a594-9f6fe37ab4fd None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:05.630097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-05e3a10b-2196-47c9-a594-9f6fe37ab4fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:05.690096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea91592-4472-458b-8dcc-350ab89914f9 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:05.712416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:05.719674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 100/186] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:05.719674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14c3561-87d6-4634-9cee-b4de6127e714 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.719902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14c3561-87d6-4634-9cee-b4de6127e714 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.720096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14c3561-87d6-4634-9cee-b4de6127e714 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.739273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0211220a-d448-480e-a636-29fe91743bc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:05.739606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d14c3561-87d6-4634-9cee-b4de6127e714 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:05.739891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0211220a-d448-480e-a636-29fe91743bc3 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.740172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0211220a-d448-480e-a636-29fe91743bc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.740441 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0211220a-d448-480e-a636-29fe91743bc3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.740787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d14c3561-87d6-4634-9cee-b4de6127e714 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:05.741164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0211220a-d448-480e-a636-29fe91743bc3 None admin] RBAC: Adding query filter params (role.id=e4fa1b2d8c404336a855b6142fbf25d4, scope.project.id=449b012a1fc24571bd474f23d9c1f706, user.id=d7ae9bf858ce45f1887ed9fbf2036581) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:05.741467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0211220a-d448-480e-a636-29fe91743bc3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:05.742943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0211220a-d448-480e-a636-29fe91743bc3 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:05.750595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 87/187] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:05 2022] GET /identity/v3/role_assignments?user.id=d7ae9bf858ce45f1887ed9fbf2036581&scope.project.id=449b012a1fc24571bd474f23d9c1f706&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 608 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:05.751432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0211220a-d448-480e-a636-29fe91743bc3 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:05.870174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0211220a-d448-480e-a636-29fe91743bc3 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:05.909794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf91f04-3195-46b6-89d0-45ca2dd28e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:05.910823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf91f04-3195-46b6-89d0-45ca2dd28e0e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:05.911608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf91f04-3195-46b6-89d0-45ca2dd28e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:05.912165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf91f04-3195-46b6-89d0-45ca2dd28e0e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:05.924901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf91f04-3195-46b6-89d0-45ca2dd28e0e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:05.930502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 101/188] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-ecf91f04-3195-46b6-89d0-45ca2dd28e0e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:05.931065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:05.931926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:05.932705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:05.933844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14c3561-87d6-4634-9cee-b4de6127e714 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:05.936797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 88/189] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:05.948036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf91f04-3195-46b6-89d0-45ca2dd28e0e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.014576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:06.027393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:06.065609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14c3561-87d6-4634-9cee-b4de6127e714 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.097867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf91f04-3195-46b6-89d0-45ca2dd28e0e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.109411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29059642-e1b3-498c-970a-37d12b0e79d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.110046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29059642-e1b3-498c-970a-37d12b0e79d3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.110691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29059642-e1b3-498c-970a-37d12b0e79d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.110969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29059642-e1b3-498c-970a-37d12b0e79d3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.124227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29059642-e1b3-498c-970a-37d12b0e79d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-29059642-e1b3-498c-970a-37d12b0e79d3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:06.129092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:06.129961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:06.130923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:06.131949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 102/190] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:41:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:06.132690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29059642-e1b3-498c-970a-37d12b0e79d3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.145759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb38da0-5a40-42b2-98a0-b4b493a6fc95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.148929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb38da0-5a40-42b2-98a0-b4b493a6fc95 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.148929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb38da0-5a40-42b2-98a0-b4b493a6fc95 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.148929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb38da0-5a40-42b2-98a0-b4b493a6fc95 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.148929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb38da0-5a40-42b2-98a0-b4b493a6fc95 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:06.148929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb38da0-5a40-42b2-98a0-b4b493a6fc95 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:06.149629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb38da0-5a40-42b2-98a0-b4b493a6fc95 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:06.159966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 89/191] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:06.160886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb38da0-5a40-42b2-98a0-b4b493a6fc95 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.247203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29059642-e1b3-498c-970a-37d12b0e79d3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.267175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb38da0-5a40-42b2-98a0-b4b493a6fc95 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.269250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:06.271456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:06.283091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f50e65-223e-4b5f-9e6f-eb684d79c532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.283713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f50e65-223e-4b5f-9e6f-eb684d79c532 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.283985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f50e65-223e-4b5f-9e6f-eb684d79c532 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.284249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f50e65-223e-4b5f-9e6f-eb684d79c532 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.294726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f50e65-223e-4b5f-9e6f-eb684d79c532 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a4f50e65-223e-4b5f-9e6f-eb684d79c532 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:06.299511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:06.300371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:06.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 103/192] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:06.301806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f50e65-223e-4b5f-9e6f-eb684d79c532 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.304616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe7e279-7fe1-4251-9578-11789cd7345d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.305487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe7e279-7fe1-4251-9578-11789cd7345d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.305754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe7e279-7fe1-4251-9578-11789cd7345d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.306854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe7e279-7fe1-4251-9578-11789cd7345d None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.307515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe7e279-7fe1-4251-9578-11789cd7345d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:06.307818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe7e279-7fe1-4251-9578-11789cd7345d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:06.309754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe7e279-7fe1-4251-9578-11789cd7345d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:06.317826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 90/193] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:06.318657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe7e279-7fe1-4251-9578-11789cd7345d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.397751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f50e65-223e-4b5f-9e6f-eb684d79c532 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.420414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe7e279-7fe1-4251-9578-11789cd7345d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.431214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab27c70-ba77-4eab-a517-b153cd042a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.431860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab27c70-ba77-4eab-a517-b153cd042a57 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.432136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab27c70-ba77-4eab-a517-b153cd042a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.432407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab27c70-ba77-4eab-a517-b153cd042a57 None admin] PATH_INFO: `/v3/users/cdb53d400a754eca8b13d369ed0bc0ab` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.444946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab27c70-ba77-4eab-a517-b153cd042a57 None admin] RBAC: Authorizing `identity:get_user(user_id=cdb53d400a754eca8b13d369ed0bc0ab)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:06.446882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab27c70-ba77-4eab-a517-b153cd042a57 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:06.448615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 104/194] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:06 2022] GET /identity/v3/users/cdb53d400a754eca8b13d369ed0bc0ab => generated 283 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:06.449237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab27c70-ba77-4eab-a517-b153cd042a57 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.453389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea38d9eb-ab73-4913-9f02-f2518161012b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.454094 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea38d9eb-ab73-4913-9f02-f2518161012b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.454372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea38d9eb-ab73-4913-9f02-f2518161012b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.454638 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea38d9eb-ab73-4913-9f02-f2518161012b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.455724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea38d9eb-ab73-4913-9f02-f2518161012b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:06.457128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea38d9eb-ab73-4913-9f02-f2518161012b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:06.462482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 91/195] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:06.462482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea38d9eb-ab73-4913-9f02-f2518161012b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.490130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:06.495735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:06.550910 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab27c70-ba77-4eab-a517-b153cd042a57 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.560114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea38d9eb-ab73-4913-9f02-f2518161012b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.584455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1280426-7d69-4462-ac9d-8fc96a527cca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.585050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1280426-7d69-4462-ac9d-8fc96a527cca None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.585327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1280426-7d69-4462-ac9d-8fc96a527cca None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.585596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1280426-7d69-4462-ac9d-8fc96a527cca None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.586273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1280426-7d69-4462-ac9d-8fc96a527cca None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:06.586569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1280426-7d69-4462-ac9d-8fc96a527cca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:06.587962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1280426-7d69-4462-ac9d-8fc96a527cca None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:06.592977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa53cd70-689a-4621-a050-aa5bfb709a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.593597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa53cd70-689a-4621-a050-aa5bfb709a20 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.593875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa53cd70-689a-4621-a050-aa5bfb709a20 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.594173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa53cd70-689a-4621-a050-aa5bfb709a20 None admin] PATH_INFO: `/v3/users/cdb53d400a754eca8b13d369ed0bc0ab` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.595286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa53cd70-689a-4621-a050-aa5bfb709a20 None admin] RBAC: Authorizing `identity:get_user(user_id=cdb53d400a754eca8b13d369ed0bc0ab)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:06.595712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 105/196] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:06 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:06.596490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1280426-7d69-4462-ac9d-8fc96a527cca None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.596739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa53cd70-689a-4621-a050-aa5bfb709a20 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:06.598443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 92/197] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:06 2022] GET /identity/v3/users/cdb53d400a754eca8b13d369ed0bc0ab => generated 283 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:06.599043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa53cd70-689a-4621-a050-aa5bfb709a20 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.695529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:06.699150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1280426-7d69-4462-ac9d-8fc96a527cca None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.699320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa53cd70-689a-4621-a050-aa5bfb709a20 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.701369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:06.732301 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef88de8f-6158-4632-a34d-90ed6f345583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.732554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d39b9d-b692-4605-b300-0e581253edbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.732895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef88de8f-6158-4632-a34d-90ed6f345583 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.733153 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d39b9d-b692-4605-b300-0e581253edbf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.733244 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef88de8f-6158-4632-a34d-90ed6f345583 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.733425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef88de8f-6158-4632-a34d-90ed6f345583 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.733511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d39b9d-b692-4605-b300-0e581253edbf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.733688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d39b9d-b692-4605-b300-0e581253edbf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.734093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef88de8f-6158-4632-a34d-90ed6f345583 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:06.735505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef88de8f-6158-4632-a34d-90ed6f345583 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:06.762481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 93/198] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:41:06 2022] POST /identity/v3/users => generated 297 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:41:06.763298 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef88de8f-6158-4632-a34d-90ed6f345583 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.807935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d39b9d-b692-4605-b300-0e581253edbf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:06.809297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d39b9d-b692-4605-b300-0e581253edbf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:06.879428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef88de8f-6158-4632-a34d-90ed6f345583 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:06.931041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 106/199] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:06.931231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6abaed-e5eb-48c3-bd99-01d48b739a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:06.931339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d39b9d-b692-4605-b300-0e581253edbf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.931450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6abaed-e5eb-48c3-bd99-01d48b739a79 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:06.931450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6abaed-e5eb-48c3-bd99-01d48b739a79 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:06.931450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6abaed-e5eb-48c3-bd99-01d48b739a79 None admin] PATH_INFO: `/v3/users/d7c226f9fa6c4d79867811e016a8115e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:06.931450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6abaed-e5eb-48c3-bd99-01d48b739a79 None admin] RBAC: Authorizing `identity:get_user(user_id=d7c226f9fa6c4d79867811e016a8115e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:06.931450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6abaed-e5eb-48c3-bd99-01d48b739a79 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:06.932749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 94/200] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:06 2022] GET /identity/v3/users/d7c226f9fa6c4d79867811e016a8115e => generated 289 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:06.936179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:06.938080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6abaed-e5eb-48c3-bd99-01d48b739a79 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:06.965967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:07.056914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d39b9d-b692-4605-b300-0e581253edbf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:07.057534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6abaed-e5eb-48c3-bd99-01d48b739a79 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:07.101352 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603f61b3-6fa7-4013-8ae8-116090ccf4eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:07.101907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603f61b3-6fa7-4013-8ae8-116090ccf4eb None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:07.102214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603f61b3-6fa7-4013-8ae8-116090ccf4eb None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:07.102391 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7defeb-1a58-454b-ab39-f6090ec78232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:07.102514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603f61b3-6fa7-4013-8ae8-116090ccf4eb None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:07.102994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7defeb-1a58-454b-ab39-f6090ec78232 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:07.103277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7defeb-1a58-454b-ab39-f6090ec78232 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:07.103549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7defeb-1a58-454b-ab39-f6090ec78232 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:07.103649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603f61b3-6fa7-4013-8ae8-116090ccf4eb None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:07.105074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603f61b3-6fa7-4013-8ae8-116090ccf4eb None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:07.107107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 95/201] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:06 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:07.108773 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603f61b3-6fa7-4013-8ae8-116090ccf4eb None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:07.180802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7defeb-1a58-454b-ab39-f6090ec78232 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:07.183510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7defeb-1a58-454b-ab39-f6090ec78232 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:07.186467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:07.226549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:07.234494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603f61b3-6fa7-4013-8ae8-116090ccf4eb None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:07.287590 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6184508-3bfa-420e-bcc8-d7d3e816ece3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:07.288735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6184508-3bfa-420e-bcc8-d7d3e816ece3 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:07.289221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6184508-3bfa-420e-bcc8-d7d3e816ece3 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:07.289867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6184508-3bfa-420e-bcc8-d7d3e816ece3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:07.301569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 107/202] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:07.301569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7defeb-1a58-454b-ab39-f6090ec78232 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:07.390553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6184508-3bfa-420e-bcc8-d7d3e816ece3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:07.392104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6184508-3bfa-420e-bcc8-d7d3e816ece3 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:07.421271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7defeb-1a58-454b-ab39-f6090ec78232 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:07.435320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:07.468593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03361c4f-97ab-4fe9-b320-6b41c9fad02f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:07.469569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03361c4f-97ab-4fe9-b320-6b41c9fad02f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:07.470188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03361c4f-97ab-4fe9-b320-6b41c9fad02f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:07.470743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03361c4f-97ab-4fe9-b320-6b41c9fad02f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:07.471663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03361c4f-97ab-4fe9-b320-6b41c9fad02f None admin] RBAC: Adding query filter params (role.id=5e19a16e5bfd4c94b1c34ae717ccfc42, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa, user.id=cdb53d400a754eca8b13d369ed0bc0ab) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:07.472683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03361c4f-97ab-4fe9-b320-6b41c9fad02f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:07.474799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03361c4f-97ab-4fe9-b320-6b41c9fad02f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:07.482944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 108/203] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:07 2022] GET /identity/v3/role_assignments?user.id=cdb53d400a754eca8b13d369ed0bc0ab&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:07.484117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03361c4f-97ab-4fe9-b320-6b41c9fad02f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:07.507542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:07.531190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 96/204] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:07.538613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6184508-3bfa-420e-bcc8-d7d3e816ece3 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:07.649505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03361c4f-97ab-4fe9-b320-6b41c9fad02f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:07.692949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bdc46d-fdbc-4d94-a0a6-c6ab1a8affa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:07.694886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdc46d-fdbc-4d94-a0a6-c6ab1a8affa2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:07.696719 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdc46d-fdbc-4d94-a0a6-c6ab1a8affa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:07.697210 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bdc46d-fdbc-4d94-a0a6-c6ab1a8affa2 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:07.700525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bdc46d-fdbc-4d94-a0a6-c6ab1a8affa2 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:07.702520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07bdc46d-fdbc-4d94-a0a6-c6ab1a8affa2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:07.704378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 109/205] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:07 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:07.711203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bdc46d-fdbc-4d94-a0a6-c6ab1a8affa2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:07.759319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6184508-3bfa-420e-bcc8-d7d3e816ece3 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:07.766512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:07.806745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09edfe0-d351-452a-867d-69e41bdcefae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:07.807754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09edfe0-d351-452a-867d-69e41bdcefae None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:07.808403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09edfe0-d351-452a-867d-69e41bdcefae None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:07.811357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09edfe0-d351-452a-867d-69e41bdcefae None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:07.814047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09edfe0-d351-452a-867d-69e41bdcefae None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:07.815720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09edfe0-d351-452a-867d-69e41bdcefae None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:07.817539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 97/206] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:07 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:07.827923 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09edfe0-d351-452a-867d-69e41bdcefae None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:07.894717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bdc46d-fdbc-4d94-a0a6-c6ab1a8affa2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:07.945131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:07.970789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fb341e-1faa-41a5-b2a0-0fdc17f475b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:07.971415 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fb341e-1faa-41a5-b2a0-0fdc17f475b6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:07.971697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fb341e-1faa-41a5-b2a0-0fdc17f475b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:07.971973 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fb341e-1faa-41a5-b2a0-0fdc17f475b6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:07.972708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fb341e-1faa-41a5-b2a0-0fdc17f475b6 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa, user.id=cdb53d400a754eca8b13d369ed0bc0ab) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:07.973016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fb341e-1faa-41a5-b2a0-0fdc17f475b6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:07.974489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fb341e-1faa-41a5-b2a0-0fdc17f475b6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:07.986367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 110/207] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:07 2022] GET /identity/v3/role_assignments?user.id=cdb53d400a754eca8b13d369ed0bc0ab&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 259 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:08.003616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09edfe0-d351-452a-867d-69e41bdcefae None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:08.057406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2d80fd-4393-412c-93ab-c7ef6e7c61f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:08.058943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2d80fd-4393-412c-93ab-c7ef6e7c61f6 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:08.059473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2d80fd-4393-412c-93ab-c7ef6e7c61f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:08.059942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2d80fd-4393-412c-93ab-c7ef6e7c61f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:08.061086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2d80fd-4393-412c-93ab-c7ef6e7c61f6 None admin] RBAC: Adding query filter params (role.id=680bd58399004020984edb53af05d3f8, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e, user.id=d7c226f9fa6c4d79867811e016a8115e) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:08.061534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2d80fd-4393-412c-93ab-c7ef6e7c61f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:08.063025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2d80fd-4393-412c-93ab-c7ef6e7c61f6 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:08.070016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 98/208] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:07 2022] GET /identity/v3/role_assignments?user.id=d7c226f9fa6c4d79867811e016a8115e&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=680bd58399004020984edb53af05d3f8 => generated 608 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:09.213559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c37a313-5822-4998-925a-ee5c595e137e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:09.214181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c37a313-5822-4998-925a-ee5c595e137e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:09.214181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c37a313-5822-4998-925a-ee5c595e137e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:09.215532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 111/209] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:09.220863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7cdc62-9d90-4862-a394-cb3e114d9036 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:09.221239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7cdc62-9d90-4862-a394-cb3e114d9036 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:09.221419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7cdc62-9d90-4862-a394-cb3e114d9036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:09.240315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-1c7cdc62-9d90-4862-a394-cb3e114d9036 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:09.241516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1c7cdc62-9d90-4862-a394-cb3e114d9036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:09.268881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ebcaa0-21db-4dd2-94a8-0069dbd61b30 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:09.269169 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ebcaa0-21db-4dd2-94a8-0069dbd61b30 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:09.269435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ebcaa0-21db-4dd2-94a8-0069dbd61b30 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:09.270985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 112/210] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:09.277810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd37300-d2c2-48e5-a824-618e6c52a0a7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:09.278100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd37300-d2c2-48e5-a824-618e6c52a0a7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:09.278377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd37300-d2c2-48e5-a824-618e6c52a0a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:09.297266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3dd37300-d2c2-48e5-a824-618e6c52a0a7 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:09.298429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3dd37300-d2c2-48e5-a824-618e6c52a0a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:09.318004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 99/211] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:09.371515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 113/212] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:09.586270 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7cdc62-9d90-4862-a394-cb3e114d9036 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:09.643796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:09.647843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd37300-d2c2-48e5-a824-618e6c52a0a7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:09.678793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:09.700931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7cdc62-9d90-4862-a394-cb3e114d9036 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:09.742049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66970e92-8d58-404a-ac9a-0b6dfef8f4f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:09.742722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66970e92-8d58-404a-ac9a-0b6dfef8f4f5 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:09.743055 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66970e92-8d58-404a-ac9a-0b6dfef8f4f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:09.743260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66970e92-8d58-404a-ac9a-0b6dfef8f4f5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:09.745175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66970e92-8d58-404a-ac9a-0b6dfef8f4f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:09.747648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66970e92-8d58-404a-ac9a-0b6dfef8f4f5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:09.749726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 100/213] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:09.755763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66970e92-8d58-404a-ac9a-0b6dfef8f4f5 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:09.758427 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd37300-d2c2-48e5-a824-618e6c52a0a7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:09.795446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e92cb0-f1b0-4d47-a93e-6ed7d00ac7a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:09.797323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e92cb0-f1b0-4d47-a93e-6ed7d00ac7a3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:09.797808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e92cb0-f1b0-4d47-a93e-6ed7d00ac7a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:09.798302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e92cb0-f1b0-4d47-a93e-6ed7d00ac7a3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:09.812184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e92cb0-f1b0-4d47-a93e-6ed7d00ac7a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-35e92cb0-f1b0-4d47-a93e-6ed7d00ac7a3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:09.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:09.819497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:09.820292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:09.828274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 114/214] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:09.831943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e92cb0-f1b0-4d47-a93e-6ed7d00ac7a3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:09.883656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66970e92-8d58-404a-ac9a-0b6dfef8f4f5 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:09.912970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:09.928352 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e31320d-daec-4a1b-8462-5a8a1fbeb11c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:09.929293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e31320d-daec-4a1b-8462-5a8a1fbeb11c None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:09.929765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e31320d-daec-4a1b-8462-5a8a1fbeb11c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:09.930284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e31320d-daec-4a1b-8462-5a8a1fbeb11c None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:09.931135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e31320d-daec-4a1b-8462-5a8a1fbeb11c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:09.933527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e31320d-daec-4a1b-8462-5a8a1fbeb11c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:09.949664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:09.963103 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 101/215] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:41:09 2022] POST /identity/v3/users => generated 297 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:41:09.963609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e92cb0-f1b0-4d47-a93e-6ed7d00ac7a3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:09.965233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef3601b-bf9a-4bd6-aa1c-cd7de32446b5 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:09.965802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef3601b-bf9a-4bd6-aa1c-cd7de32446b5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:09.966353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef3601b-bf9a-4bd6-aa1c-cd7de32446b5 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:09.968024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 102/216] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:09.974120 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dad7ba-2988-4f97-80c1-c30f5ed70b33 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:09.974429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dad7ba-2988-4f97-80c1-c30f5ed70b33 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:09.978087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dad7ba-2988-4f97-80c1-c30f5ed70b33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:09.995362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-08dad7ba-2988-4f97-80c1-c30f5ed70b33 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:09.996813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-08dad7ba-2988-4f97-80c1-c30f5ed70b33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:10.005004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227ce58a-39e9-4797-9948-84c466b77abd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:10.005625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227ce58a-39e9-4797-9948-84c466b77abd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:10.006014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227ce58a-39e9-4797-9948-84c466b77abd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:10.006321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227ce58a-39e9-4797-9948-84c466b77abd None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:10.007001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227ce58a-39e9-4797-9948-84c466b77abd None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:10.007304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227ce58a-39e9-4797-9948-84c466b77abd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:10.008897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227ce58a-39e9-4797-9948-84c466b77abd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:10.019983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 115/217] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:10.031355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227ce58a-39e9-4797-9948-84c466b77abd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:10.085577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 103/218] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:10.153248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227ce58a-39e9-4797-9948-84c466b77abd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:10.192311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6bd888-788b-4e84-98ae-55fa23975310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:10.193340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6bd888-788b-4e84-98ae-55fa23975310 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:10.194002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6bd888-788b-4e84-98ae-55fa23975310 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:10.194739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6bd888-788b-4e84-98ae-55fa23975310 None admin] PATH_INFO: `/v3/users/68528ceb502e4f70b41c8a1ccebf2d65` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:10.212168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6bd888-788b-4e84-98ae-55fa23975310 None admin] RBAC: Authorizing `identity:get_user(user_id=68528ceb502e4f70b41c8a1ccebf2d65)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:10.214017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6bd888-788b-4e84-98ae-55fa23975310 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:10.217040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 116/219] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:10 2022] GET /identity/v3/users/68528ceb502e4f70b41c8a1ccebf2d65 => generated 297 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:10.224551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dad7ba-2988-4f97-80c1-c30f5ed70b33 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:10.304785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f533e919-bf71-4d7d-88ad-38208c2894c0 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:10.305064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f533e919-bf71-4d7d-88ad-38208c2894c0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:10.305335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f533e919-bf71-4d7d-88ad-38208c2894c0 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:10.306911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 117/220] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:10.312999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a97582b-37d8-405c-b383-eeec11e60b70 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:10.313353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a97582b-37d8-405c-b383-eeec11e60b70 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:10.313714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a97582b-37d8-405c-b383-eeec11e60b70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:10.335306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9a97582b-37d8-405c-b383-eeec11e60b70 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:10.336697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9a97582b-37d8-405c-b383-eeec11e60b70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:10.340512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:10.349937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dad7ba-2988-4f97-80c1-c30f5ed70b33 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:10.360606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:10.394405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4adb6f-81e7-496e-adc3-03566d24140d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:10.395026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4adb6f-81e7-496e-adc3-03566d24140d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:10.395305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4adb6f-81e7-496e-adc3-03566d24140d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:10.395577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4adb6f-81e7-496e-adc3-03566d24140d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:10.426429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 118/221] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:10.430835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a97582b-37d8-405c-b383-eeec11e60b70 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:10.479003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4adb6f-81e7-496e-adc3-03566d24140d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:10.480340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4adb6f-81e7-496e-adc3-03566d24140d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:10.567243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a97582b-37d8-405c-b383-eeec11e60b70 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:10.603494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 104/222] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:10.611832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4549e292-ebdf-4f9d-b938-5d3c8d22f895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:10.612123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4adb6f-81e7-496e-adc3-03566d24140d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:10.612483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4549e292-ebdf-4f9d-b938-5d3c8d22f895 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:10.612761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4549e292-ebdf-4f9d-b938-5d3c8d22f895 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:10.613033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4549e292-ebdf-4f9d-b938-5d3c8d22f895 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:10.626761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:10.630575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:10.637222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4549e292-ebdf-4f9d-b938-5d3c8d22f895 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4549e292-ebdf-4f9d-b938-5d3c8d22f895 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:10.643099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:10.644034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:10.644966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 119/223] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:10.653067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4549e292-ebdf-4f9d-b938-5d3c8d22f895 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:10.771188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4549e292-ebdf-4f9d-b938-5d3c8d22f895 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:10.781476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4adb6f-81e7-496e-adc3-03566d24140d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:10.812493 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09424ea9-6344-4166-ab12-7d8523b7d4bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:10.813255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09424ea9-6344-4166-ab12-7d8523b7d4bf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:10.813533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09424ea9-6344-4166-ab12-7d8523b7d4bf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:10.813802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09424ea9-6344-4166-ab12-7d8523b7d4bf None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:10.814790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09424ea9-6344-4166-ab12-7d8523b7d4bf None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:10.815101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09424ea9-6344-4166-ab12-7d8523b7d4bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:10.816737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09424ea9-6344-4166-ab12-7d8523b7d4bf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:10.824117 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eff8cb-3adb-4088-9119-40e0f24e200d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:10.825132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eff8cb-3adb-4088-9119-40e0f24e200d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:10.825609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eff8cb-3adb-4088-9119-40e0f24e200d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:10.826028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 120/224] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:10.826521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06eff8cb-3adb-4088-9119-40e0f24e200d None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:10.826714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09424ea9-6344-4166-ab12-7d8523b7d4bf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:10.828309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06eff8cb-3adb-4088-9119-40e0f24e200d None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:10.829931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06eff8cb-3adb-4088-9119-40e0f24e200d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:10.832279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 105/225] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:10 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:10.837832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eff8cb-3adb-4088-9119-40e0f24e200d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:10.899267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:10.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09424ea9-6344-4166-ab12-7d8523b7d4bf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:10.945622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:10.973915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06eff8cb-3adb-4088-9119-40e0f24e200d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:10.980134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7225f0c-0e69-4305-9598-a95ddf33911b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:10.980969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7225f0c-0e69-4305-9598-a95ddf33911b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:10.981437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7225f0c-0e69-4305-9598-a95ddf33911b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:10.981898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7225f0c-0e69-4305-9598-a95ddf33911b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:10.992889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7225f0c-0e69-4305-9598-a95ddf33911b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a7225f0c-0e69-4305-9598-a95ddf33911b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:10.999838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:11.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:11.001559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:11.003957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 121/226] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:41:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:11.004925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7225f0c-0e69-4305-9598-a95ddf33911b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:11.017467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad9be39-db53-4132-82c3-1076062641a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:11.018114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad9be39-db53-4132-82c3-1076062641a5 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:11.018393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad9be39-db53-4132-82c3-1076062641a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:11.018660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad9be39-db53-4132-82c3-1076062641a5 None admin] PATH_INFO: `/v3/users/cdb53d400a754eca8b13d369ed0bc0ab` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:11.019883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad9be39-db53-4132-82c3-1076062641a5 None admin] RBAC: Authorizing `identity:get_user(user_id=cdb53d400a754eca8b13d369ed0bc0ab)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:11.021373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad9be39-db53-4132-82c3-1076062641a5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:11.023892 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 106/227] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:10 2022] GET /identity/v3/users/cdb53d400a754eca8b13d369ed0bc0ab => generated 283 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:11.024778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad9be39-db53-4132-82c3-1076062641a5 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:11.125097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7225f0c-0e69-4305-9598-a95ddf33911b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:11.144667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad9be39-db53-4132-82c3-1076062641a5 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:11.148957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:11.162191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7886bd8-4d96-42a2-8a74-106863f93d07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:11.163126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7886bd8-4d96-42a2-8a74-106863f93d07 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:11.163594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7886bd8-4d96-42a2-8a74-106863f93d07 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:11.164075 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7886bd8-4d96-42a2-8a74-106863f93d07 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:11.165002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7886bd8-4d96-42a2-8a74-106863f93d07 None admin] RBAC: Adding query filter params (role.id=e4fa1b2d8c404336a855b6142fbf25d4, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa, user.id=68528ceb502e4f70b41c8a1ccebf2d65) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:11.165556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7886bd8-4d96-42a2-8a74-106863f93d07 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:11.167364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7886bd8-4d96-42a2-8a74-106863f93d07 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:11.173566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 122/228] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:11 2022] GET /identity/v3/role_assignments?user.id=68528ceb502e4f70b41c8a1ccebf2d65&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:11.183940 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7886bd8-4d96-42a2-8a74-106863f93d07 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:11.189248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e400a98-45ca-4cc3-8de1-673ac92c9c4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:11.189667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e400a98-45ca-4cc3-8de1-673ac92c9c4c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:11.189942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e400a98-45ca-4cc3-8de1-673ac92c9c4c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:11.190255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e400a98-45ca-4cc3-8de1-673ac92c9c4c None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:11.190948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e400a98-45ca-4cc3-8de1-673ac92c9c4c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:11.191247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e400a98-45ca-4cc3-8de1-673ac92c9c4c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:11.192664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e400a98-45ca-4cc3-8de1-673ac92c9c4c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:11.202656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 107/229] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:11.212572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e400a98-45ca-4cc3-8de1-673ac92c9c4c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:11.225382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:11.363133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7886bd8-4d96-42a2-8a74-106863f93d07 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:11.403691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59360152-5fea-4bfb-add4-0ddb08b1ad07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:11.404319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59360152-5fea-4bfb-add4-0ddb08b1ad07 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:11.404589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59360152-5fea-4bfb-add4-0ddb08b1ad07 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:11.404855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59360152-5fea-4bfb-add4-0ddb08b1ad07 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:11.405963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59360152-5fea-4bfb-add4-0ddb08b1ad07 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:11.407789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59360152-5fea-4bfb-add4-0ddb08b1ad07 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:11.409396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 123/230] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:11 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:11.415687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59360152-5fea-4bfb-add4-0ddb08b1ad07 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:11.452771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e400a98-45ca-4cc3-8de1-673ac92c9c4c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:11.475085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:11.532701 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59360152-5fea-4bfb-add4-0ddb08b1ad07 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:11.535890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc1f8e7-b717-47ce-97d4-8a083bfa00ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:11.536531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc1f8e7-b717-47ce-97d4-8a083bfa00ff None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:11.536809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc1f8e7-b717-47ce-97d4-8a083bfa00ff None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:11.537081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc1f8e7-b717-47ce-97d4-8a083bfa00ff None admin] PATH_INFO: `/v3/users/cdb53d400a754eca8b13d369ed0bc0ab` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:11.539015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc1f8e7-b717-47ce-97d4-8a083bfa00ff None admin] RBAC: Authorizing `identity:get_user(user_id=cdb53d400a754eca8b13d369ed0bc0ab)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:11.540443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc1f8e7-b717-47ce-97d4-8a083bfa00ff None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:11.547755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 108/231] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:11 2022] GET /identity/v3/users/cdb53d400a754eca8b13d369ed0bc0ab => generated 283 bytes in 340 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:11.553393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc1f8e7-b717-47ce-97d4-8a083bfa00ff None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:11.629792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea089bb6-e980-4c0c-84fb-eddd22ef8a56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:11.630414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea089bb6-e980-4c0c-84fb-eddd22ef8a56 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:11.630692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea089bb6-e980-4c0c-84fb-eddd22ef8a56 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:11.630962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea089bb6-e980-4c0c-84fb-eddd22ef8a56 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/users/cdb53d400a754eca8b13d369ed0bc0ab/roles/5e19a16e5bfd4c94b1c34ae717ccfc42` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:11.636548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea089bb6-e980-4c0c-84fb-eddd22ef8a56 None admin] RBAC: Authorizing `identity:create_grant(project_id=74b104206ad84d6bbb9d5fd926b6f1aa, user_id=cdb53d400a754eca8b13d369ed0bc0ab, role_id=5e19a16e5bfd4c94b1c34ae717ccfc42)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:11.638111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea089bb6-e980-4c0c-84fb-eddd22ef8a56 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:11.644995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:11.667203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 124/232] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:41:11 2022] PUT /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/users/cdb53d400a754eca8b13d369ed0bc0ab/roles/5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:41:11.674448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc1f8e7-b717-47ce-97d4-8a083bfa00ff None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:11.712754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9248f9e2-6e4e-4529-8b53-94b8b5b2915f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:11.713413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9248f9e2-6e4e-4529-8b53-94b8b5b2915f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:11.713691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9248f9e2-6e4e-4529-8b53-94b8b5b2915f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:11.714577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9248f9e2-6e4e-4529-8b53-94b8b5b2915f None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:11.715896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9248f9e2-6e4e-4529-8b53-94b8b5b2915f None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:11.717370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9248f9e2-6e4e-4529-8b53-94b8b5b2915f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:11.719110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 109/233] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:11 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:11.724965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea089bb6-e980-4c0c-84fb-eddd22ef8a56 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:11.840279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea089bb6-e980-4c0c-84fb-eddd22ef8a56 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:11.855616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:11.878543 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20eb88d-1cf4-47e2-b4b7-4fdbe4f01d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:11.879514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20eb88d-1cf4-47e2-b4b7-4fdbe4f01d3e None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:11.880068 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20eb88d-1cf4-47e2-b4b7-4fdbe4f01d3e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:11.880514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20eb88d-1cf4-47e2-b4b7-4fdbe4f01d3e None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/users/cdb53d400a754eca8b13d369ed0bc0ab/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:11.882710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20eb88d-1cf4-47e2-b4b7-4fdbe4f01d3e None admin] RBAC: Authorizing `identity:create_grant(project_id=74b104206ad84d6bbb9d5fd926b6f1aa, user_id=cdb53d400a754eca8b13d369ed0bc0ab, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:11.884427 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20eb88d-1cf4-47e2-b4b7-4fdbe4f01d3e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:11.907255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 125/234] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:41:11 2022] PUT /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/users/cdb53d400a754eca8b13d369ed0bc0ab/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:41:12.634733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8044d11f-b6ae-4632-8b00-198e71df6198 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:12.635923 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8044d11f-b6ae-4632-8b00-198e71df6198 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:12.636218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8044d11f-b6ae-4632-8b00-198e71df6198 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:12.637955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 110/235] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:12.643658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900a763d-8ea7-47c0-bd63-a5dff051e2d5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:12.644173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900a763d-8ea7-47c0-bd63-a5dff051e2d5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:12.644272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900a763d-8ea7-47c0-bd63-a5dff051e2d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:12.668805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-900a763d-8ea7-47c0-bd63-a5dff051e2d5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:12.670041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-900a763d-8ea7-47c0-bd63-a5dff051e2d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:12.746979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 126/236] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:12.770833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae982c5-4b07-4b4e-a8cc-5266592ceca8 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:12.771152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae982c5-4b07-4b4e-a8cc-5266592ceca8 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:12.771386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae982c5-4b07-4b4e-a8cc-5266592ceca8 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:12.772990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 111/237] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:12.778132 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33028916-f25b-4cc8-a2e6-99f1edfff3a1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:12.778632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33028916-f25b-4cc8-a2e6-99f1edfff3a1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:12.779118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33028916-f25b-4cc8-a2e6-99f1edfff3a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:12.803349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-33028916-f25b-4cc8-a2e6-99f1edfff3a1 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:12.804879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-33028916-f25b-4cc8-a2e6-99f1edfff3a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:12.896576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 127/238] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:13.060395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae982c5-4b07-4b4e-a8cc-5266592ceca8 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:13.191730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae982c5-4b07-4b4e-a8cc-5266592ceca8 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:13.228272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33028916-f25b-4cc8-a2e6-99f1edfff3a1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:13.243258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4605d4d1-536c-4f3a-a700-2672c60ace2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:13.245893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4605d4d1-536c-4f3a-a700-2672c60ace2a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:13.246593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4605d4d1-536c-4f3a-a700-2672c60ace2a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:13.247694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4605d4d1-536c-4f3a-a700-2672c60ace2a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:13.269363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4605d4d1-536c-4f3a-a700-2672c60ace2a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-4605d4d1-536c-4f3a-a700-2672c60ace2a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:13.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:13.280433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:13.281227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:13.286627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 112/239] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:13 2022] GET /identity/v3/roles/reader => generated 84 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:13.292764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:13.294525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4605d4d1-536c-4f3a-a700-2672c60ace2a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:13.302478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:13.383628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33028916-f25b-4cc8-a2e6-99f1edfff3a1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:13.416660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4605d4d1-536c-4f3a-a700-2672c60ace2a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:13.452641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c1bb6e-6504-4f44-bc1e-4c95d4da4228 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:13.452837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e25835-5889-4e45-aa2a-618ade5373a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:13.453192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c1bb6e-6504-4f44-bc1e-4c95d4da4228 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:13.453467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c1bb6e-6504-4f44-bc1e-4c95d4da4228 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:13.453736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c1bb6e-6504-4f44-bc1e-4c95d4da4228 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:13.454253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e25835-5889-4e45-aa2a-618ade5373a8 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:13.454705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e25835-5889-4e45-aa2a-618ade5373a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:13.455161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e25835-5889-4e45-aa2a-618ade5373a8 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:13.455991 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e25835-5889-4e45-aa2a-618ade5373a8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:13.456467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e25835-5889-4e45-aa2a-618ade5373a8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:13.458004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e25835-5889-4e45-aa2a-618ade5373a8 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:13.458705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c1bb6e-6504-4f44-bc1e-4c95d4da4228 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:13.460152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c1bb6e-6504-4f44-bc1e-4c95d4da4228 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:13.461715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 128/240] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:13.469072 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c1bb6e-6504-4f44-bc1e-4c95d4da4228 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:13.472182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 113/241] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:13 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:13.477986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e25835-5889-4e45-aa2a-618ade5373a8 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:13.598596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:13.650838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c1bb6e-6504-4f44-bc1e-4c95d4da4228 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:13.653668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:13.673476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e25835-5889-4e45-aa2a-618ade5373a8 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:13.702803 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3f99dc-b61f-45cd-8c60-24abc6bf5fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:13.703741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3f99dc-b61f-45cd-8c60-24abc6bf5fb5 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:13.704223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3f99dc-b61f-45cd-8c60-24abc6bf5fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:13.704686 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3f99dc-b61f-45cd-8c60-24abc6bf5fb5 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:13.706029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3f99dc-b61f-45cd-8c60-24abc6bf5fb5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:13.707726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3f99dc-b61f-45cd-8c60-24abc6bf5fb5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:13.734414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 129/242] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:41:13 2022] POST /identity/v3/users => generated 293 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:41:13.746840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a00936a-417e-4173-bfdd-a225a40abf12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:13.747495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a00936a-417e-4173-bfdd-a225a40abf12 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:13.747777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a00936a-417e-4173-bfdd-a225a40abf12 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:13.748111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a00936a-417e-4173-bfdd-a225a40abf12 None admin] PATH_INFO: `/v3/users/b6c03f5a18484d73bd61e54ffc0ce3eb` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:13.775466 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a00936a-417e-4173-bfdd-a225a40abf12 None admin] RBAC: Authorizing `identity:get_user(user_id=b6c03f5a18484d73bd61e54ffc0ce3eb)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:13.777062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a00936a-417e-4173-bfdd-a225a40abf12 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:13.779212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 114/243] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:13 2022] GET /identity/v3/users/b6c03f5a18484d73bd61e54ffc0ce3eb => generated 297 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:13.784973 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3f99dc-b61f-45cd-8c60-24abc6bf5fb5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:13.910878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3f99dc-b61f-45cd-8c60-24abc6bf5fb5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:13.955197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcf1630-134d-4388-b0ef-87407599bda9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:13.956167 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcf1630-134d-4388-b0ef-87407599bda9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:13.956646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcf1630-134d-4388-b0ef-87407599bda9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:13.957111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcf1630-134d-4388-b0ef-87407599bda9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:13.986692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:14.015116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa7ad8f-be4f-423b-a440-fbfb7f95349e None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:14.015421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa7ad8f-be4f-423b-a440-fbfb7f95349e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:14.015659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa7ad8f-be4f-423b-a440-fbfb7f95349e None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:14.016990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 115/244] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:41:14.022492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8082635f-7510-4751-9778-8c3679dca1ab None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:14.022833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8082635f-7510-4751-9778-8c3679dca1ab None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:14.023050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8082635f-7510-4751-9778-8c3679dca1ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:14.047621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8082635f-7510-4751-9778-8c3679dca1ab None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:14.049072 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8082635f-7510-4751-9778-8c3679dca1ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:14.055386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcf1630-134d-4388-b0ef-87407599bda9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:14.056893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcf1630-134d-4388-b0ef-87407599bda9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:14.130319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 116/245] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:14.178905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 130/246] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:14.187404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8082635f-7510-4751-9778-8c3679dca1ab None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:14.248805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:14.304647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8082635f-7510-4751-9778-8c3679dca1ab None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:14.346197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52bda4a-6876-4c24-9d10-2cdc8bd2d1a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:14.346834 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52bda4a-6876-4c24-9d10-2cdc8bd2d1a0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:14.347111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52bda4a-6876-4c24-9d10-2cdc8bd2d1a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:14.347433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52bda4a-6876-4c24-9d10-2cdc8bd2d1a0 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:14.348580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52bda4a-6876-4c24-9d10-2cdc8bd2d1a0 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:14.350016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52bda4a-6876-4c24-9d10-2cdc8bd2d1a0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:14.357023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 117/247] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:14 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:14.358364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcf1630-134d-4388-b0ef-87407599bda9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:14.419607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52bda4a-6876-4c24-9d10-2cdc8bd2d1a0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:14.423665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:14.476193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcf1630-134d-4388-b0ef-87407599bda9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:14.541215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d7442f-0fe2-4e50-962b-79abecd62b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:14.542754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d7442f-0fe2-4e50-962b-79abecd62b12 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:14.543174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d7442f-0fe2-4e50-962b-79abecd62b12 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:14.543873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d7442f-0fe2-4e50-962b-79abecd62b12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:14.545234 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d7442f-0fe2-4e50-962b-79abecd62b12 None admin] RBAC: Adding query filter params (role.id=680bd58399004020984edb53af05d3f8, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa, user.id=b6c03f5a18484d73bd61e54ffc0ce3eb) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:14.545724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d7442f-0fe2-4e50-962b-79abecd62b12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:14.547744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d7442f-0fe2-4e50-962b-79abecd62b12 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:14.552998 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52bda4a-6876-4c24-9d10-2cdc8bd2d1a0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:14.557299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 131/248] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:14 2022] GET /identity/v3/role_assignments?user.id=b6c03f5a18484d73bd61e54ffc0ce3eb&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=680bd58399004020984edb53af05d3f8 => generated 259 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:14.580645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:14.600015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c186a56f-8877-49ed-8708-72b919508599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:14.600672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c186a56f-8877-49ed-8708-72b919508599 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:14.600949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c186a56f-8877-49ed-8708-72b919508599 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:14.601275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c186a56f-8877-49ed-8708-72b919508599 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:14.614274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c186a56f-8877-49ed-8708-72b919508599 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-c186a56f-8877-49ed-8708-72b919508599 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:14.620319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:14.621296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:14.622201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 118/249] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:14.628374 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d7442f-0fe2-4e50-962b-79abecd62b12 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:14.656351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ee679b-4894-4edc-a136-1222f36a4743 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:14.656681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ee679b-4894-4edc-a136-1222f36a4743 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:14.656909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ee679b-4894-4edc-a136-1222f36a4743 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:14.659157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 119/250] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:41:14.665548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f731c15-3644-401e-8678-10fc25f675f9 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:14.665829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f731c15-3644-401e-8678-10fc25f675f9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:14.666132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f731c15-3644-401e-8678-10fc25f675f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:14.689933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-6f731c15-3644-401e-8678-10fc25f675f9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:14.691183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6f731c15-3644-401e-8678-10fc25f675f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:14.779135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d7442f-0fe2-4e50-962b-79abecd62b12 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:14.794862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 120/251] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:14.810699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:14.812056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc56e40-ae71-4138-aafb-6cf65e2b90e2 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:14.812355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc56e40-ae71-4138-aafb-6cf65e2b90e2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:14.812957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc56e40-ae71-4138-aafb-6cf65e2b90e2 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:14.814840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 121/252] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:41:14.821853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82487374-797f-4669-a6dd-40fd4bcad6a3 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:14.822174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82487374-797f-4669-a6dd-40fd4bcad6a3 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:14.822455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82487374-797f-4669-a6dd-40fd4bcad6a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:14.825472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d967ec-e9fc-4110-8568-c4748b862819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:14.826108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d967ec-e9fc-4110-8568-c4748b862819 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:14.826390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d967ec-e9fc-4110-8568-c4748b862819 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:14.826666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d967ec-e9fc-4110-8568-c4748b862819 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:14.827334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d967ec-e9fc-4110-8568-c4748b862819 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:14.827636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d967ec-e9fc-4110-8568-c4748b862819 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:14.829069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d967ec-e9fc-4110-8568-c4748b862819 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:14.842314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 132/253] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:14 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:14.845776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-82487374-797f-4669-a6dd-40fd4bcad6a3 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:14.847392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-82487374-797f-4669-a6dd-40fd4bcad6a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:14.848500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d967ec-e9fc-4110-8568-c4748b862819 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:14.958743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 122/254] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:15.012542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d967ec-e9fc-4110-8568-c4748b862819 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:15.064037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f86da4-9039-4ecb-8cd9-1f11c2eca2b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:15.064599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f86da4-9039-4ecb-8cd9-1f11c2eca2b7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:15.064876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f86da4-9039-4ecb-8cd9-1f11c2eca2b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:15.065146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f86da4-9039-4ecb-8cd9-1f11c2eca2b7 None admin] PATH_INFO: `/v3/users/68528ceb502e4f70b41c8a1ccebf2d65` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:15.066351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f86da4-9039-4ecb-8cd9-1f11c2eca2b7 None admin] RBAC: Authorizing `identity:get_user(user_id=68528ceb502e4f70b41c8a1ccebf2d65)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:15.067799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f86da4-9039-4ecb-8cd9-1f11c2eca2b7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:15.069768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 133/255] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:14 2022] GET /identity/v3/users/68528ceb502e4f70b41c8a1ccebf2d65 => generated 297 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:15.075875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82487374-797f-4669-a6dd-40fd4bcad6a3 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:15.119239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:15.151895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f86da4-9039-4ecb-8cd9-1f11c2eca2b7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:15.180937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82487374-797f-4669-a6dd-40fd4bcad6a3 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:15.208122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:15.220101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0262f9-fa4f-4ae6-8a35-371291839de8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:15.220962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0262f9-fa4f-4ae6-8a35-371291839de8 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:15.221410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0262f9-fa4f-4ae6-8a35-371291839de8 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:15.221867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0262f9-fa4f-4ae6-8a35-371291839de8 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:15.223294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0262f9-fa4f-4ae6-8a35-371291839de8 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:15.224888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0262f9-fa4f-4ae6-8a35-371291839de8 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:15.226916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 123/256] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:15 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:15.233514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0262f9-fa4f-4ae6-8a35-371291839de8 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:15.258352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f86da4-9039-4ecb-8cd9-1f11c2eca2b7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:15.294258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda8b101-1929-4d21-af41-4b336a4a5039 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:15.294892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda8b101-1929-4d21-af41-4b336a4a5039 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:15.295169 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda8b101-1929-4d21-af41-4b336a4a5039 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:15.295435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda8b101-1929-4d21-af41-4b336a4a5039 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:15.309196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda8b101-1929-4d21-af41-4b336a4a5039 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-cda8b101-1929-4d21-af41-4b336a4a5039 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:15.316191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:15.317069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:15.317881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 134/257] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:15 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:15.323401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda8b101-1929-4d21-af41-4b336a4a5039 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:15.373515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0262f9-fa4f-4ae6-8a35-371291839de8 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:15.394492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:15.449113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda8b101-1929-4d21-af41-4b336a4a5039 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:15.457316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e4b320-64bc-42c1-aaff-a17700a009d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:15.458316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e4b320-64bc-42c1-aaff-a17700a009d1 None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:15.458895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e4b320-64bc-42c1-aaff-a17700a009d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:15.459371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e4b320-64bc-42c1-aaff-a17700a009d1 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/users/68528ceb502e4f70b41c8a1ccebf2d65/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:15.462896 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:15.466094 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e4b320-64bc-42c1-aaff-a17700a009d1 None admin] RBAC: Authorizing `identity:create_grant(project_id=74b104206ad84d6bbb9d5fd926b6f1aa, user_id=68528ceb502e4f70b41c8a1ccebf2d65, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:15.467807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e4b320-64bc-42c1-aaff-a17700a009d1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:15.488676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5ef2ac-9237-4c1e-8191-54356d857f15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:15.489690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5ef2ac-9237-4c1e-8191-54356d857f15 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:15.490129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5ef2ac-9237-4c1e-8191-54356d857f15 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:15.490688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5ef2ac-9237-4c1e-8191-54356d857f15 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:15.491537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5ef2ac-9237-4c1e-8191-54356d857f15 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:15.491979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5ef2ac-9237-4c1e-8191-54356d857f15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:15.493494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 124/258] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:41:15 2022] PUT /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/users/68528ceb502e4f70b41c8a1ccebf2d65/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:41:15.493992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5ef2ac-9237-4c1e-8191-54356d857f15 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:15.494387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e4b320-64bc-42c1-aaff-a17700a009d1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:15.517531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 135/259] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:15.523906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5ef2ac-9237-4c1e-8191-54356d857f15 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:15.698589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e4b320-64bc-42c1-aaff-a17700a009d1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:15.718239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5ef2ac-9237-4c1e-8191-54356d857f15 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:15.754810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0648ee-5416-4715-a098-782d3ef04628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:15.755470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0648ee-5416-4715-a098-782d3ef04628 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:15.755741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0648ee-5416-4715-a098-782d3ef04628 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:15.756012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0648ee-5416-4715-a098-782d3ef04628 None admin] PATH_INFO: `/v3/users/cdb53d400a754eca8b13d369ed0bc0ab` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:15.757136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0648ee-5416-4715-a098-782d3ef04628 None admin] RBAC: Authorizing `identity:get_user(user_id=cdb53d400a754eca8b13d369ed0bc0ab)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:15.758685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0648ee-5416-4715-a098-782d3ef04628 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:15.761592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99ee1c-a544-48f7-a5aa-986e3f94db80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:15.761737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 136/260] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:15 2022] GET /identity/v3/users/cdb53d400a754eca8b13d369ed0bc0ab => generated 283 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:15.767393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0648ee-5416-4715-a098-782d3ef04628 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:15.767573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99ee1c-a544-48f7-a5aa-986e3f94db80 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:15.768179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99ee1c-a544-48f7-a5aa-986e3f94db80 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:15.768699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99ee1c-a544-48f7-a5aa-986e3f94db80 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:15.781395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d99ee1c-a544-48f7-a5aa-986e3f94db80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-6d99ee1c-a544-48f7-a5aa-986e3f94db80 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:15.791234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:15.792767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:15.793551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:15.796134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 125/261] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:41:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:15.804049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99ee1c-a544-48f7-a5aa-986e3f94db80 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:15.822106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:15.847346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:15.875221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0648ee-5416-4715-a098-782d3ef04628 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:15.922763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd783ef-7474-4e6e-ad74-9037e1e2e772 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:15.923381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd783ef-7474-4e6e-ad74-9037e1e2e772 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:15.923661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd783ef-7474-4e6e-ad74-9037e1e2e772 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:15.924143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd783ef-7474-4e6e-ad74-9037e1e2e772 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:15.933261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99ee1c-a544-48f7-a5aa-986e3f94db80 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:15.970350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0133a7-4521-4634-99c3-0119a64f7378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:15.971367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0133a7-4521-4634-99c3-0119a64f7378 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:15.971848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0133a7-4521-4634-99c3-0119a64f7378 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:15.972320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0133a7-4521-4634-99c3-0119a64f7378 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:15.973173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0133a7-4521-4634-99c3-0119a64f7378 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:15.973667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0133a7-4521-4634-99c3-0119a64f7378 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:15.975422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0133a7-4521-4634-99c3-0119a64f7378 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:15.985288 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 126/262] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:15.991294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0133a7-4521-4634-99c3-0119a64f7378 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:16.002797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd783ef-7474-4e6e-ad74-9037e1e2e772 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:16.004297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd783ef-7474-4e6e-ad74-9037e1e2e772 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:16.058780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:16.097691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:16.111361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0133a7-4521-4634-99c3-0119a64f7378 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:16.116445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 137/263] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:16.122512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd783ef-7474-4e6e-ad74-9037e1e2e772 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:16.151422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768630a5-489e-48aa-ba37-39e178b62f02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:16.152409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768630a5-489e-48aa-ba37-39e178b62f02 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:16.152982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768630a5-489e-48aa-ba37-39e178b62f02 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:16.153454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768630a5-489e-48aa-ba37-39e178b62f02 None admin] PATH_INFO: `/v3/users/cdb53d400a754eca8b13d369ed0bc0ab` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:16.154989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768630a5-489e-48aa-ba37-39e178b62f02 None admin] RBAC: Authorizing `identity:get_user(user_id=cdb53d400a754eca8b13d369ed0bc0ab)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:16.156737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768630a5-489e-48aa-ba37-39e178b62f02 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:16.159087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 127/264] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:15 2022] GET /identity/v3/users/cdb53d400a754eca8b13d369ed0bc0ab => generated 283 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:16.165076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768630a5-489e-48aa-ba37-39e178b62f02 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:16.224672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd783ef-7474-4e6e-ad74-9037e1e2e772 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:16.258890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d218a5c-c0b3-4847-9b46-2efeeb79de9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:16.259517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d218a5c-c0b3-4847-9b46-2efeeb79de9e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:16.259792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d218a5c-c0b3-4847-9b46-2efeeb79de9e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:16.260349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d218a5c-c0b3-4847-9b46-2efeeb79de9e None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:16.262288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d218a5c-c0b3-4847-9b46-2efeeb79de9e None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:16.263731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d218a5c-c0b3-4847-9b46-2efeeb79de9e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:16.265625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 138/265] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:16 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:16.272169 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d218a5c-c0b3-4847-9b46-2efeeb79de9e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:16.287405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:16.306001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768630a5-489e-48aa-ba37-39e178b62f02 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:16.376330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7058ee41-2096-444e-9fb4-2af71690c361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:16.379099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7058ee41-2096-444e-9fb4-2af71690c361 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:16.380722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7058ee41-2096-444e-9fb4-2af71690c361 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:16.381626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7058ee41-2096-444e-9fb4-2af71690c361 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:16.453054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:16.483920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d218a5c-c0b3-4847-9b46-2efeeb79de9e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:16.534896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7058ee41-2096-444e-9fb4-2af71690c361 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:16.536537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7058ee41-2096-444e-9fb4-2af71690c361 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:16.553617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950072d2-a370-43e2-86f5-b71ddef34fbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:16.554275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950072d2-a370-43e2-86f5-b71ddef34fbc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950072d2-a370-43e2-86f5-b71ddef34fbc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:16.554826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950072d2-a370-43e2-86f5-b71ddef34fbc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:16.555551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950072d2-a370-43e2-86f5-b71ddef34fbc None admin] RBAC: Adding query filter params (role.id=5e19a16e5bfd4c94b1c34ae717ccfc42, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa, user.id=cdb53d400a754eca8b13d369ed0bc0ab) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:16.555857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950072d2-a370-43e2-86f5-b71ddef34fbc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:16.557424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-950072d2-a370-43e2-86f5-b71ddef34fbc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:16.569649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 139/266] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:16 2022] GET /identity/v3/role_assignments?user.id=cdb53d400a754eca8b13d369ed0bc0ab&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 608 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:16.672768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 128/267] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:16.679080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950072d2-a370-43e2-86f5-b71ddef34fbc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:16.775029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:16.809570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950072d2-a370-43e2-86f5-b71ddef34fbc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:16.853771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f833c82f-4b47-4dea-8b47-5de02f22a9c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:16.854515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f833c82f-4b47-4dea-8b47-5de02f22a9c3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:16.854797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f833c82f-4b47-4dea-8b47-5de02f22a9c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:16.855066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f833c82f-4b47-4dea-8b47-5de02f22a9c3 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:16.857263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f833c82f-4b47-4dea-8b47-5de02f22a9c3 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:16.858759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f833c82f-4b47-4dea-8b47-5de02f22a9c3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:16.860467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 140/268] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:16 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:16.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7058ee41-2096-444e-9fb4-2af71690c361 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:16.933876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:16.983755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7058ee41-2096-444e-9fb4-2af71690c361 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:17.039635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58205e2c-508d-4d89-8de4-5b78ea98400c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:17.040248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58205e2c-508d-4d89-8de4-5b78ea98400c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:17.040522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58205e2c-508d-4d89-8de4-5b78ea98400c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:17.040791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58205e2c-508d-4d89-8de4-5b78ea98400c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:17.041579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58205e2c-508d-4d89-8de4-5b78ea98400c None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa, user.id=cdb53d400a754eca8b13d369ed0bc0ab) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:17.041930 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58205e2c-508d-4d89-8de4-5b78ea98400c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:17.043409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58205e2c-508d-4d89-8de4-5b78ea98400c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:17.054848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 129/269] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:16 2022] GET /identity/v3/role_assignments?user.id=cdb53d400a754eca8b13d369ed0bc0ab&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:17.692141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b38e34e-3421-4762-bfb9-ab060ebc9467 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:17.692512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b38e34e-3421-4762-bfb9-ab060ebc9467 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:17.692703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b38e34e-3421-4762-bfb9-ab060ebc9467 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:17.694225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 141/270] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:17.699811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c7dd19-615b-4ac8-a3d5-d3dae6f5b436 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:17.700327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c7dd19-615b-4ac8-a3d5-d3dae6f5b436 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:17.700799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c7dd19-615b-4ac8-a3d5-d3dae6f5b436 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:17.721202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a9c7dd19-615b-4ac8-a3d5-d3dae6f5b436 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:17.722928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a9c7dd19-615b-4ac8-a3d5-d3dae6f5b436 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:17.801823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 130/271] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:17.849334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27cc5a1-9deb-42f6-9eb4-8a0a3cb822b5 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:17.849835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27cc5a1-9deb-42f6-9eb4-8a0a3cb822b5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:17.850283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27cc5a1-9deb-42f6-9eb4-8a0a3cb822b5 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:17.851847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 142/272] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:17.857928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368fdd8a-81ec-4237-bd48-eac1e6a129ae None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:17.858485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368fdd8a-81ec-4237-bd48-eac1e6a129ae None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:17.858963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368fdd8a-81ec-4237-bd48-eac1e6a129ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:17.862943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a29ab7-3aa0-4707-b558-30e48fb1f99b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:17.863759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a29ab7-3aa0-4707-b558-30e48fb1f99b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:17.864260 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a29ab7-3aa0-4707-b558-30e48fb1f99b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:17.865693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 143/273] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:41:17.871397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8342d-dc46-49c2-a6c8-25014b7d94c3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:17.871922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8342d-dc46-49c2-a6c8-25014b7d94c3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:17.872271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d8342d-dc46-49c2-a6c8-25014b7d94c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:17.884416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-368fdd8a-81ec-4237-bd48-eac1e6a129ae None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:17.886548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-368fdd8a-81ec-4237-bd48-eac1e6a129ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:17.893301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-47d8342d-dc46-49c2-a6c8-25014b7d94c3 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:17.894798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-47d8342d-dc46-49c2-a6c8-25014b7d94c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:17.944827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:17.967129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 144/274] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:17.968399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 131/275] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:18.102293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d8342d-dc46-49c2-a6c8-25014b7d94c3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:18.175006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:18.205823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d8342d-dc46-49c2-a6c8-25014b7d94c3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:18.240357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368fdd8a-81ec-4237-bd48-eac1e6a129ae None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:18.240697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494dcaa6-f4c4-4d4d-b7c6-cdb0fa7a1590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:18.241429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494dcaa6-f4c4-4d4d-b7c6-cdb0fa7a1590 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:18.241707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494dcaa6-f4c4-4d4d-b7c6-cdb0fa7a1590 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:18.241978 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494dcaa6-f4c4-4d4d-b7c6-cdb0fa7a1590 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:18.252631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494dcaa6-f4c4-4d4d-b7c6-cdb0fa7a1590 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-494dcaa6-f4c4-4d4d-b7c6-cdb0fa7a1590 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:18.257460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:18.258375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:18.259185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 145/276] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:18.267083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494dcaa6-f4c4-4d4d-b7c6-cdb0fa7a1590 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:18.347088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368fdd8a-81ec-4237-bd48-eac1e6a129ae None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:18.379713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494dcaa6-f4c4-4d4d-b7c6-cdb0fa7a1590 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:18.382876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c9833c-2fb2-4222-ad7b-2483d61d52b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:18.383709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c9833c-2fb2-4222-ad7b-2483d61d52b0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:18.384180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c9833c-2fb2-4222-ad7b-2483d61d52b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:18.384639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c9833c-2fb2-4222-ad7b-2483d61d52b0 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:18.399068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c9833c-2fb2-4222-ad7b-2483d61d52b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-08c9833c-2fb2-4222-ad7b-2483d61d52b0 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:18.405374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:18.406622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:18.407607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:18.408801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 132/277] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:18.408801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c9833c-2fb2-4222-ad7b-2483d61d52b0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:18.418220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:18.424634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4de283d-26d1-4ca7-b73c-e7764368980d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:18.425240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de283d-26d1-4ca7-b73c-e7764368980d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:18.425519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de283d-26d1-4ca7-b73c-e7764368980d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:18.425793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4de283d-26d1-4ca7-b73c-e7764368980d None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:18.426484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4de283d-26d1-4ca7-b73c-e7764368980d None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:18.426793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4de283d-26d1-4ca7-b73c-e7764368980d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:18.428169 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4de283d-26d1-4ca7-b73c-e7764368980d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:18.436738 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 146/278] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:18.437514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4de283d-26d1-4ca7-b73c-e7764368980d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:18.458928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:18.515477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c9833c-2fb2-4222-ad7b-2483d61d52b0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:18.534321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4de283d-26d1-4ca7-b73c-e7764368980d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:18.549389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3550aaae-e668-4742-88da-559816dec5cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:18.550015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3550aaae-e668-4742-88da-559816dec5cd None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:18.550327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3550aaae-e668-4742-88da-559816dec5cd None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:18.550604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3550aaae-e668-4742-88da-559816dec5cd None admin] PATH_INFO: `/v3/domains/default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:18.551705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3550aaae-e668-4742-88da-559816dec5cd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:18.553130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3550aaae-e668-4742-88da-559816dec5cd None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:18.554711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 133/279] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:18.559310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c51f287-f567-4570-a293-84ac888683ae None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:18.559591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c51f287-f567-4570-a293-84ac888683ae None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:18.559870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c51f287-f567-4570-a293-84ac888683ae None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:18.561124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 134/280] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:18 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:18.561817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c51f287-f567-4570-a293-84ac888683ae None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:18.568358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d696953-c55b-4fe2-8ecf-6fd8921f9ac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:18.568962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d696953-c55b-4fe2-8ecf-6fd8921f9ac1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:18.569235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d696953-c55b-4fe2-8ecf-6fd8921f9ac1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:18.569505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d696953-c55b-4fe2-8ecf-6fd8921f9ac1 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:18.570186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d696953-c55b-4fe2-8ecf-6fd8921f9ac1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:18.570486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d696953-c55b-4fe2-8ecf-6fd8921f9ac1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:18.571845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d696953-c55b-4fe2-8ecf-6fd8921f9ac1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:18.579849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 147/281] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:18 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:18.580594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d696953-c55b-4fe2-8ecf-6fd8921f9ac1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:18.634604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:18.666273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c51f287-f567-4570-a293-84ac888683ae None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:18.676678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:18.680577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d696953-c55b-4fe2-8ecf-6fd8921f9ac1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:18.700642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b076f56-3d5f-466e-9086-809d49574bec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:18.701263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b076f56-3d5f-466e-9086-809d49574bec None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:18.701542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b076f56-3d5f-466e-9086-809d49574bec None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:18.701812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b076f56-3d5f-466e-9086-809d49574bec None admin] PATH_INFO: `/v3/users/863367eeb7ab443ca8c269f2f5e8e58c` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:18.713557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f813f4df-6a3e-4e7d-8da6-24db4a3f4232 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:18.714118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f813f4df-6a3e-4e7d-8da6-24db4a3f4232 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:18.714401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f813f4df-6a3e-4e7d-8da6-24db4a3f4232 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:18.714674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f813f4df-6a3e-4e7d-8da6-24db4a3f4232 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:18.714785 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b076f56-3d5f-466e-9086-809d49574bec None admin] RBAC: Authorizing `identity:get_user(user_id=863367eeb7ab443ca8c269f2f5e8e58c)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:18.715311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f813f4df-6a3e-4e7d-8da6-24db4a3f4232 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:18.716145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b076f56-3d5f-466e-9086-809d49574bec None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:18.716722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f813f4df-6a3e-4e7d-8da6-24db4a3f4232 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:18.722439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 135/282] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:18 2022] GET /identity/v3/users/863367eeb7ab443ca8c269f2f5e8e58c => generated 293 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:18.722439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb92e15-ae8d-4448-b7eb-4931fc9ba86b None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:18.722439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb92e15-ae8d-4448-b7eb-4931fc9ba86b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:18.722439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb92e15-ae8d-4448-b7eb-4931fc9ba86b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:18.738269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-eeb92e15-ae8d-4448-b7eb-4931fc9ba86b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:18.739434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-eeb92e15-ae8d-4448-b7eb-4931fc9ba86b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:18.740386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 148/283] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:41:18 2022] POST /identity/v3/users => generated 293 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:41:18.741371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f813f4df-6a3e-4e7d-8da6-24db4a3f4232 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:18.824955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 136/284] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:18.826082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb92e15-ae8d-4448-b7eb-4931fc9ba86b None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:18.871931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f813f4df-6a3e-4e7d-8da6-24db4a3f4232 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:18.888536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:18.919525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253fd0b3-1495-4d26-ac68-3f4f165ffb67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:18.920168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253fd0b3-1495-4d26-ac68-3f4f165ffb67 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:18.920445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253fd0b3-1495-4d26-ac68-3f4f165ffb67 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:18.920718 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253fd0b3-1495-4d26-ac68-3f4f165ffb67 None admin] PATH_INFO: `/v3/users/b6c03f5a18484d73bd61e54ffc0ce3eb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:18.921846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253fd0b3-1495-4d26-ac68-3f4f165ffb67 None admin] RBAC: Authorizing `identity:get_user(user_id=b6c03f5a18484d73bd61e54ffc0ce3eb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:18.923573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253fd0b3-1495-4d26-ac68-3f4f165ffb67 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:18.925187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 149/285] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:18 2022] GET /identity/v3/users/b6c03f5a18484d73bd61e54ffc0ce3eb => generated 297 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:18.931015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253fd0b3-1495-4d26-ac68-3f4f165ffb67 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:18.958495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:18.964062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb92e15-ae8d-4448-b7eb-4931fc9ba86b None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:19.019295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab4a13d-e2ed-4469-b91a-d543a8312a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:19.020149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab4a13d-e2ed-4469-b91a-d543a8312a1c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:19.020599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab4a13d-e2ed-4469-b91a-d543a8312a1c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:19.021031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab4a13d-e2ed-4469-b91a-d543a8312a1c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:19.022220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab4a13d-e2ed-4469-b91a-d543a8312a1c None admin] RBAC: Adding query filter params (role.id=e4fa1b2d8c404336a855b6142fbf25d4, user.id=863367eeb7ab443ca8c269f2f5e8e58c, scope.system=all) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:19.022551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab4a13d-e2ed-4469-b91a-d543a8312a1c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:19.024385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab4a13d-e2ed-4469-b91a-d543a8312a1c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:19.033523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 137/286] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Oct 2 00:41:18 2022] GET /identity/v3/role_assignments?user.id=863367eeb7ab443ca8c269f2f5e8e58c&scope.system=all&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 226 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:19.058748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253fd0b3-1495-4d26-ac68-3f4f165ffb67 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:19.105599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dc56cd-bc52-43e2-8c3f-88beaee1be14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:19.106384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dc56cd-bc52-43e2-8c3f-88beaee1be14 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:19.106663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dc56cd-bc52-43e2-8c3f-88beaee1be14 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:19.106933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dc56cd-bc52-43e2-8c3f-88beaee1be14 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:19.109085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dc56cd-bc52-43e2-8c3f-88beaee1be14 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:19.110598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dc56cd-bc52-43e2-8c3f-88beaee1be14 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:19.112300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 150/287] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:18 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:19.118652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab4a13d-e2ed-4469-b91a-d543a8312a1c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:19.124833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dc56cd-bc52-43e2-8c3f-88beaee1be14 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:19.189298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:19.254445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dc56cd-bc52-43e2-8c3f-88beaee1be14 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:19.255287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab4a13d-e2ed-4469-b91a-d543a8312a1c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:19.297259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de679de-e0e3-412f-8a4a-e9bc7f65f801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:19.297893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de679de-e0e3-412f-8a4a-e9bc7f65f801 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:19.298198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de679de-e0e3-412f-8a4a-e9bc7f65f801 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:19.298467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de679de-e0e3-412f-8a4a-e9bc7f65f801 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:19.301927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86df43d7-2297-43de-9089-f865e1731a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:19.302634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86df43d7-2297-43de-9089-f865e1731a08 None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:19.302913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86df43d7-2297-43de-9089-f865e1731a08 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:19.303188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86df43d7-2297-43de-9089-f865e1731a08 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/users/b6c03f5a18484d73bd61e54ffc0ce3eb/roles/680bd58399004020984edb53af05d3f8` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:19.305571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86df43d7-2297-43de-9089-f865e1731a08 None admin] RBAC: Authorizing `identity:create_grant(project_id=74b104206ad84d6bbb9d5fd926b6f1aa, user_id=b6c03f5a18484d73bd61e54ffc0ce3eb, role_id=680bd58399004020984edb53af05d3f8)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:19.308234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86df43d7-2297-43de-9089-f865e1731a08 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:19.310220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de679de-e0e3-412f-8a4a-e9bc7f65f801 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5de679de-e0e3-412f-8a4a-e9bc7f65f801 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:19.316010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:19.316904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:19.317695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 151/288] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:19.323514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de679de-e0e3-412f-8a4a-e9bc7f65f801 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:19.323743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 138/289] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:41:19 2022] PUT /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/users/b6c03f5a18484d73bd61e54ffc0ce3eb/roles/680bd58399004020984edb53af05d3f8 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:41:19.465094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de679de-e0e3-412f-8a4a-e9bc7f65f801 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:19.476978 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:19.512679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7d7b2e-7bb8-49fd-9d4d-879c41b9884c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:19.513213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d7b2e-7bb8-49fd-9d4d-879c41b9884c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:19.513485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d7b2e-7bb8-49fd-9d4d-879c41b9884c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:19.513755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7d7b2e-7bb8-49fd-9d4d-879c41b9884c None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:19.514534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7d7b2e-7bb8-49fd-9d4d-879c41b9884c None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:19.514837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7d7b2e-7bb8-49fd-9d4d-879c41b9884c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:19.516293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7d7b2e-7bb8-49fd-9d4d-879c41b9884c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:19.529460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 152/290] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:19.535160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86df43d7-2297-43de-9089-f865e1731a08 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:19.580528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:19.673586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86df43d7-2297-43de-9089-f865e1731a08 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:19.710847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38661983-3c64-40a5-beec-625f223b8ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:19.711558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38661983-3c64-40a5-beec-625f223b8ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:19.711840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38661983-3c64-40a5-beec-625f223b8ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:19.712110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38661983-3c64-40a5-beec-625f223b8ce7 None admin] PATH_INFO: `/v3/users/68528ceb502e4f70b41c8a1ccebf2d65` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:19.713308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38661983-3c64-40a5-beec-625f223b8ce7 None admin] RBAC: Authorizing `identity:get_user(user_id=68528ceb502e4f70b41c8a1ccebf2d65)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:19.714778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38661983-3c64-40a5-beec-625f223b8ce7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:19.716465 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 139/291] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:19 2022] GET /identity/v3/users/68528ceb502e4f70b41c8a1ccebf2d65 => generated 297 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:19.722264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7d7b2e-7bb8-49fd-9d4d-879c41b9884c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:19.851480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7d7b2e-7bb8-49fd-9d4d-879c41b9884c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:19.891708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae8438e-f0cd-46f8-9e28-e24b780dbe33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:19.892316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae8438e-f0cd-46f8-9e28-e24b780dbe33 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:19.892583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae8438e-f0cd-46f8-9e28-e24b780dbe33 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:19.892897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae8438e-f0cd-46f8-9e28-e24b780dbe33 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:19.933699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:19.973686 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae8438e-f0cd-46f8-9e28-e24b780dbe33 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:19.975170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae8438e-f0cd-46f8-9e28-e24b780dbe33 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:20.090651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 153/292] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:20.096652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38661983-3c64-40a5-beec-625f223b8ce7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:20.198489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:20.209277 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38661983-3c64-40a5-beec-625f223b8ce7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:20.243898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99cf2f6-cda7-4d5c-b294-6f293be4335c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:20.244815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99cf2f6-cda7-4d5c-b294-6f293be4335c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:20.245299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99cf2f6-cda7-4d5c-b294-6f293be4335c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:20.245775 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99cf2f6-cda7-4d5c-b294-6f293be4335c None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:20.247137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99cf2f6-cda7-4d5c-b294-6f293be4335c None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:20.248743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99cf2f6-cda7-4d5c-b294-6f293be4335c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:20.250870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 140/293] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:20 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:20.256986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae8438e-f0cd-46f8-9e28-e24b780dbe33 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:20.341748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:20.378856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae8438e-f0cd-46f8-9e28-e24b780dbe33 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:20.426816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789d157c-5348-45b4-9fa6-eb1604b4de50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:20.427440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789d157c-5348-45b4-9fa6-eb1604b4de50 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:20.427721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789d157c-5348-45b4-9fa6-eb1604b4de50 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:20.427996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789d157c-5348-45b4-9fa6-eb1604b4de50 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:20.428729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789d157c-5348-45b4-9fa6-eb1604b4de50 None admin] RBAC: Adding query filter params (role.id=e4fa1b2d8c404336a855b6142fbf25d4, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa, user.id=68528ceb502e4f70b41c8a1ccebf2d65) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:20.429038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789d157c-5348-45b4-9fa6-eb1604b4de50 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:20.430510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789d157c-5348-45b4-9fa6-eb1604b4de50 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:20.442621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 154/294] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:20 2022] GET /identity/v3/role_assignments?user.id=68528ceb502e4f70b41c8a1ccebf2d65&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:21.506844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778c173-626a-46f0-8426-e8acd2e0593b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:21.507379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778c173-626a-46f0-8426-e8acd2e0593b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:21.507379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778c173-626a-46f0-8426-e8acd2e0593b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:21.508993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 141/295] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:41:21.514714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b0cb7a-075f-4e37-89db-6f84e55248bd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:21.515113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b0cb7a-075f-4e37-89db-6f84e55248bd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:21.515297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b0cb7a-075f-4e37-89db-6f84e55248bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:21.533891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-07b0cb7a-075f-4e37-89db-6f84e55248bd None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:21.535125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-07b0cb7a-075f-4e37-89db-6f84e55248bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:21.542784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f29f4d-f08b-4bca-b8b7-0d613a27f07d None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:21.543082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f29f4d-f08b-4bca-b8b7-0d613a27f07d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:21.543353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f29f4d-f08b-4bca-b8b7-0d613a27f07d None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:21.544659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 142/296] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:41:21.551594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eeadcfc-3c26-42dd-b22a-6a6e972a98b2 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:21.551880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eeadcfc-3c26-42dd-b22a-6a6e972a98b2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:21.552154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eeadcfc-3c26-42dd-b22a-6a6e972a98b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:21.570996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-6eeadcfc-3c26-42dd-b22a-6a6e972a98b2 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:21.572177 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6eeadcfc-3c26-42dd-b22a-6a6e972a98b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:21.610297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 155/297] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:21.615263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b276cf-7eac-4829-a1ca-3c06665f3e97 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:21.615549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b276cf-7eac-4829-a1ca-3c06665f3e97 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:21.615825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b276cf-7eac-4829-a1ca-3c06665f3e97 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:21.617375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 156/298] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:21.623003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3def89af-7598-4dd5-9dc8-810fef436712 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:21.623401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3def89af-7598-4dd5-9dc8-810fef436712 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:21.623689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3def89af-7598-4dd5-9dc8-810fef436712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:21.642814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3def89af-7598-4dd5-9dc8-810fef436712 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:21.644245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3def89af-7598-4dd5-9dc8-810fef436712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:21.647065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 143/299] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:21.706668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:21.715878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 157/300] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:21.904975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eeadcfc-3c26-42dd-b22a-6a6e972a98b2 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:21.923960 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3def89af-7598-4dd5-9dc8-810fef436712 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:22.001076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:22.015027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eeadcfc-3c26-42dd-b22a-6a6e972a98b2 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.042534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3def89af-7598-4dd5-9dc8-810fef436712 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.066218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fca14c1-db14-48d5-96b1-033bdd0918d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.067162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fca14c1-db14-48d5-96b1-033bdd0918d0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.067440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fca14c1-db14-48d5-96b1-033bdd0918d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.067711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fca14c1-db14-48d5-96b1-033bdd0918d0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.068995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fca14c1-db14-48d5-96b1-033bdd0918d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:22.070503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fca14c1-db14-48d5-96b1-033bdd0918d0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:22.077282 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 144/301] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:22.077282 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fca14c1-db14-48d5-96b1-033bdd0918d0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:22.084250 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cdcda9-7e43-4434-b08e-71c63debb53c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.084875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cdcda9-7e43-4434-b08e-71c63debb53c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.085144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cdcda9-7e43-4434-b08e-71c63debb53c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.085406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cdcda9-7e43-4434-b08e-71c63debb53c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.096583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cdcda9-7e43-4434-b08e-71c63debb53c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-87cdcda9-7e43-4434-b08e-71c63debb53c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:22.101431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:22.102613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 158/302] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:21 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:22.103409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cdcda9-7e43-4434-b08e-71c63debb53c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:22.162512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:22.186773 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fca14c1-db14-48d5-96b1-033bdd0918d0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.206791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cdcda9-7e43-4434-b08e-71c63debb53c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.220379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea0e44-dc5b-4303-b1eb-635ea549d0bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.221001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea0e44-dc5b-4303-b1eb-635ea549d0bf None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.221277 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea0e44-dc5b-4303-b1eb-635ea549d0bf None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.221547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea0e44-dc5b-4303-b1eb-635ea549d0bf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.232371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ea0e44-dc5b-4303-b1eb-635ea549d0bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-39ea0e44-dc5b-4303-b1eb-635ea549d0bf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:22.237285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:22.238327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 145/303] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:22.239681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea0e44-dc5b-4303-b1eb-635ea549d0bf None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:22.241095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec61ef68-af17-4dd4-a97a-941ad3b5dec6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.241690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec61ef68-af17-4dd4-a97a-941ad3b5dec6 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.241963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec61ef68-af17-4dd4-a97a-941ad3b5dec6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.242254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec61ef68-af17-4dd4-a97a-941ad3b5dec6 None admin] PATH_INFO: `/v3/groups` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.243154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec61ef68-af17-4dd4-a97a-941ad3b5dec6 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:22.245948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec61ef68-af17-4dd4-a97a-941ad3b5dec6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:22.259554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:22.263224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 159/304] 10.222.0.25 () {60 vars in 1134 bytes} [Sun Oct 2 00:41:22 2022] POST /identity/v3/groups => generated 229 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:41:22.264485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bab6f7-b39a-4933-8395-f1247f47070a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.264773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bab6f7-b39a-4933-8395-f1247f47070a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.265043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bab6f7-b39a-4933-8395-f1247f47070a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.266436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 160/305] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:22.271317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bab6f7-b39a-4933-8395-f1247f47070a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:22.357326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea0e44-dc5b-4303-b1eb-635ea549d0bf None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.383094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bab6f7-b39a-4933-8395-f1247f47070a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.400224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:22.401655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c21fb57-34d3-461d-8b1a-3d2715c0fb1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.404013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c21fb57-34d3-461d-8b1a-3d2715c0fb1a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.404297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c21fb57-34d3-461d-8b1a-3d2715c0fb1a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.404572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c21fb57-34d3-461d-8b1a-3d2715c0fb1a None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.405275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c21fb57-34d3-461d-8b1a-3d2715c0fb1a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:22.405580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c21fb57-34d3-461d-8b1a-3d2715c0fb1a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:22.407547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c21fb57-34d3-461d-8b1a-3d2715c0fb1a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:22.425152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 146/306] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:22.425152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e040fc50-6f01-4c70-8ba8-f08391e26bbd None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.425152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e040fc50-6f01-4c70-8ba8-f08391e26bbd None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.425152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e040fc50-6f01-4c70-8ba8-f08391e26bbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.429914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe61a14c-1b1c-4e19-827e-3718d460c736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.430873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe61a14c-1b1c-4e19-827e-3718d460c736 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.431242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe61a14c-1b1c-4e19-827e-3718d460c736 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.432519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe61a14c-1b1c-4e19-827e-3718d460c736 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.433265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe61a14c-1b1c-4e19-827e-3718d460c736 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:22.433639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe61a14c-1b1c-4e19-827e-3718d460c736 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:22.435119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe61a14c-1b1c-4e19-827e-3718d460c736 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:22.443391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e040fc50-6f01-4c70-8ba8-f08391e26bbd None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:22.443830 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e040fc50-6f01-4c70-8ba8-f08391e26bbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:22.445360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 161/307] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:22.446332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe61a14c-1b1c-4e19-827e-3718d460c736 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:22.522537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 147/308] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:22.523379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e040fc50-6f01-4c70-8ba8-f08391e26bbd None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:22.534381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:22.568299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe61a14c-1b1c-4e19-827e-3718d460c736 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.614835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9caee5-f135-4716-ba58-c69f3dfe032b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.615469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9caee5-f135-4716-ba58-c69f3dfe032b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.615746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9caee5-f135-4716-ba58-c69f3dfe032b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.616017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9caee5-f135-4716-ba58-c69f3dfe032b None admin] PATH_INFO: `/v3/users/c09c126a57144a5aa89bf76276b2f0a8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.635317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e040fc50-6f01-4c70-8ba8-f08391e26bbd None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.638946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9caee5-f135-4716-ba58-c69f3dfe032b None admin] RBAC: Authorizing `identity:get_user(user_id=c09c126a57144a5aa89bf76276b2f0a8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:22.640723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9caee5-f135-4716-ba58-c69f3dfe032b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:22.648859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 162/309] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:22 2022] GET /identity/v3/users/c09c126a57144a5aa89bf76276b2f0a8 => generated 293 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:22.650363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9caee5-f135-4716-ba58-c69f3dfe032b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:22.676758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2072f194-f0b8-411d-96d2-a9fab70cde9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.677315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2072f194-f0b8-411d-96d2-a9fab70cde9b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.677597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2072f194-f0b8-411d-96d2-a9fab70cde9b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.677868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2072f194-f0b8-411d-96d2-a9fab70cde9b None admin] PATH_INFO: `/v3/users/863367eeb7ab443ca8c269f2f5e8e58c` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.679545 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2072f194-f0b8-411d-96d2-a9fab70cde9b None admin] RBAC: Authorizing `identity:get_user(user_id=863367eeb7ab443ca8c269f2f5e8e58c)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:22.681263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2072f194-f0b8-411d-96d2-a9fab70cde9b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:22.684097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 148/310] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:22 2022] GET /identity/v3/users/863367eeb7ab443ca8c269f2f5e8e58c => generated 293 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:22.689938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2072f194-f0b8-411d-96d2-a9fab70cde9b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:22.695553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:22.768493 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9caee5-f135-4716-ba58-c69f3dfe032b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.772115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:22.797924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2072f194-f0b8-411d-96d2-a9fab70cde9b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.807104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74d1dfb-715c-47a0-be8b-750fb15a1275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.807713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74d1dfb-715c-47a0-be8b-750fb15a1275 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.807985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74d1dfb-715c-47a0-be8b-750fb15a1275 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.808256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74d1dfb-715c-47a0-be8b-750fb15a1275 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.808978 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74d1dfb-715c-47a0-be8b-750fb15a1275 None admin] RBAC: Adding query filter params (role.id=680bd58399004020984edb53af05d3f8, user.id=c09c126a57144a5aa89bf76276b2f0a8, scope.system=all) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:22.809284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74d1dfb-715c-47a0-be8b-750fb15a1275 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:22.810752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74d1dfb-715c-47a0-be8b-750fb15a1275 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:22.818004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 163/311] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Oct 2 00:41:22 2022] GET /identity/v3/role_assignments?user.id=c09c126a57144a5aa89bf76276b2f0a8&scope.system=all&role.id=680bd58399004020984edb53af05d3f8 => generated 226 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:22.818684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74d1dfb-715c-47a0-be8b-750fb15a1275 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:22.838853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f3d318-23b6-4807-9641-c6d5192f4996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.840011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f3d318-23b6-4807-9641-c6d5192f4996 None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.840523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f3d318-23b6-4807-9641-c6d5192f4996 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.841104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f3d318-23b6-4807-9641-c6d5192f4996 None admin] PATH_INFO: `/v3/system/users/863367eeb7ab443ca8c269f2f5e8e58c/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:22.843132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f3d318-23b6-4807-9641-c6d5192f4996 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=863367eeb7ab443ca8c269f2f5e8e58c, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:22.845006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f3d318-23b6-4807-9641-c6d5192f4996 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:22.866166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 149/312] 10.222.0.25 () {58 vars in 1334 bytes} [Sun Oct 2 00:41:22 2022] PUT /identity/v3/system/users/863367eeb7ab443ca8c269f2f5e8e58c/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:41:22.944191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:22.950976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74d1dfb-715c-47a0-be8b-750fb15a1275 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:22.993270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e290fc18-dca6-472b-9c80-7fb138778c7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:22.993928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e290fc18-dca6-472b-9c80-7fb138778c7c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:22.994783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e290fc18-dca6-472b-9c80-7fb138778c7c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:22.995055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e290fc18-dca6-472b-9c80-7fb138778c7c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:23.007751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e290fc18-dca6-472b-9c80-7fb138778c7c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e290fc18-dca6-472b-9c80-7fb138778c7c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:23.013945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:23.014923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:23.015710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 164/313] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:23.021240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f3d318-23b6-4807-9641-c6d5192f4996 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:23.139021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f3d318-23b6-4807-9641-c6d5192f4996 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:23.175198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0579b4b-788e-4d64-aaf5-3b0a941e689e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:23.176148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0579b4b-788e-4d64-aaf5-3b0a941e689e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:23.176650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0579b4b-788e-4d64-aaf5-3b0a941e689e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:23.177154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0579b4b-788e-4d64-aaf5-3b0a941e689e None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:23.178121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0579b4b-788e-4d64-aaf5-3b0a941e689e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:23.178627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0579b4b-788e-4d64-aaf5-3b0a941e689e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:23.180199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0579b4b-788e-4d64-aaf5-3b0a941e689e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:23.189825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 150/314] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:23 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:23.201071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e290fc18-dca6-472b-9c80-7fb138778c7c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:23.307856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e290fc18-dca6-472b-9c80-7fb138778c7c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:23.343055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444c4fa6-89f9-4c12-a4c3-8b45d371de59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:23.343907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444c4fa6-89f9-4c12-a4c3-8b45d371de59 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:23.344402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444c4fa6-89f9-4c12-a4c3-8b45d371de59 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:23.344814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444c4fa6-89f9-4c12-a4c3-8b45d371de59 None admin] PATH_INFO: `/v3/users/b6c03f5a18484d73bd61e54ffc0ce3eb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:23.346123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444c4fa6-89f9-4c12-a4c3-8b45d371de59 None admin] RBAC: Authorizing `identity:get_user(user_id=b6c03f5a18484d73bd61e54ffc0ce3eb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:23.347716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444c4fa6-89f9-4c12-a4c3-8b45d371de59 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:23.349707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 165/315] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:23 2022] GET /identity/v3/users/b6c03f5a18484d73bd61e54ffc0ce3eb => generated 297 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:23.356310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0579b4b-788e-4d64-aaf5-3b0a941e689e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:23.361243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:23.462748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0579b4b-788e-4d64-aaf5-3b0a941e689e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:23.497112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5005396-ccec-4e56-8a4b-8fe7bff6872d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:23.497715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5005396-ccec-4e56-8a4b-8fe7bff6872d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:23.498036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5005396-ccec-4e56-8a4b-8fe7bff6872d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:23.498335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5005396-ccec-4e56-8a4b-8fe7bff6872d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:23.567721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5005396-ccec-4e56-8a4b-8fe7bff6872d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:23.569202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5005396-ccec-4e56-8a4b-8fe7bff6872d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:23.572380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:23.674222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 151/316] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:23.676696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444c4fa6-89f9-4c12-a4c3-8b45d371de59 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:23.707320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:23.785738 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444c4fa6-89f9-4c12-a4c3-8b45d371de59 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:23.819571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f4ac14-73d8-4ecc-ad45-5346d6ec46fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:23.820196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f4ac14-73d8-4ecc-ad45-5346d6ec46fd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:23.820470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f4ac14-73d8-4ecc-ad45-5346d6ec46fd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:23.820740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f4ac14-73d8-4ecc-ad45-5346d6ec46fd None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:23.821846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f4ac14-73d8-4ecc-ad45-5346d6ec46fd None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:23.823412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f4ac14-73d8-4ecc-ad45-5346d6ec46fd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:23.825313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 166/317] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:23 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:23.831010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5005396-ccec-4e56-8a4b-8fe7bff6872d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:23.932442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5005396-ccec-4e56-8a4b-8fe7bff6872d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:23.934083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:23.968032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b10a3a-6074-45d9-ab2a-ca9b1fb57686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:23.968633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b10a3a-6074-45d9-ab2a-ca9b1fb57686 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:23.968906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b10a3a-6074-45d9-ab2a-ca9b1fb57686 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:23.969172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b10a3a-6074-45d9-ab2a-ca9b1fb57686 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:23.969881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b10a3a-6074-45d9-ab2a-ca9b1fb57686 None admin] RBAC: Adding query filter params (role.id=680bd58399004020984edb53af05d3f8, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa, user.id=b6c03f5a18484d73bd61e54ffc0ce3eb) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:23.970210 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b10a3a-6074-45d9-ab2a-ca9b1fb57686 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:23.971687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b10a3a-6074-45d9-ab2a-ca9b1fb57686 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:23.981201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 152/318] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:23 2022] GET /identity/v3/role_assignments?user.id=b6c03f5a18484d73bd61e54ffc0ce3eb&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=680bd58399004020984edb53af05d3f8 => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:24.610280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652252e8-11c8-410c-926c-80b3d0f79fc6 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:24.610547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652252e8-11c8-410c-926c-80b3d0f79fc6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:24.610816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652252e8-11c8-410c-926c-80b3d0f79fc6 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:24.612320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 167/319] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:24.617269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc21a7-821d-40ca-9e11-64e3550383b7 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:24.617690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc21a7-821d-40ca-9e11-64e3550383b7 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:24.617844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc21a7-821d-40ca-9e11-64e3550383b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:24.637424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a6dc21a7-821d-40ca-9e11-64e3550383b7 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:24.638736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a6dc21a7-821d-40ca-9e11-64e3550383b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:24.704780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 153/320] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:24.968264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652252e8-11c8-410c-926c-80b3d0f79fc6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:25.003998 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e444c723-9cea-49f2-8d02-90cc391c6a59 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.004281 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e444c723-9cea-49f2-8d02-90cc391c6a59 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.004552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e444c723-9cea-49f2-8d02-90cc391c6a59 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.005905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 154/321] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:41:25.011362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e436ed14-69b6-441e-82e7-5d15f3f2abd5 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.011646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e436ed14-69b6-441e-82e7-5d15f3f2abd5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.011912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e436ed14-69b6-441e-82e7-5d15f3f2abd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.031362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e436ed14-69b6-441e-82e7-5d15f3f2abd5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:25.033004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e436ed14-69b6-441e-82e7-5d15f3f2abd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:25.067010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:25.076572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652252e8-11c8-410c-926c-80b3d0f79fc6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:25.106175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 155/322] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:25.116041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d4b05d-95c2-49c7-aa6d-3e13cd934cdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:25.116691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d4b05d-95c2-49c7-aa6d-3e13cd934cdb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.116962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d4b05d-95c2-49c7-aa6d-3e13cd934cdb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.117229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d4b05d-95c2-49c7-aa6d-3e13cd934cdb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.118430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d4b05d-95c2-49c7-aa6d-3e13cd934cdb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:25.120450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d4b05d-95c2-49c7-aa6d-3e13cd934cdb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:25.126627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 168/323] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:25.128264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e436ed14-69b6-441e-82e7-5d15f3f2abd5 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:25.156740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:25.229940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e436ed14-69b6-441e-82e7-5d15f3f2abd5 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:25.232446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7244a38-02fc-4f77-b8ed-c627bc2d28c7 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.232727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7244a38-02fc-4f77-b8ed-c627bc2d28c7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.233001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7244a38-02fc-4f77-b8ed-c627bc2d28c7 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.234364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 169/324] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:25.239186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d81d9-a97a-4363-8a0d-112cde389330 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.239466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d81d9-a97a-4363-8a0d-112cde389330 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.239736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9d81d9-a97a-4363-8a0d-112cde389330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.258890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ca9d81d9-a97a-4363-8a0d-112cde389330 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:25.260912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ca9d81d9-a97a-4363-8a0d-112cde389330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:25.266891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c07ab0e-a623-43cd-9885-0062d978a82e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:25.267502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c07ab0e-a623-43cd-9885-0062d978a82e None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.267780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c07ab0e-a623-43cd-9885-0062d978a82e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.268051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c07ab0e-a623-43cd-9885-0062d978a82e None admin] PATH_INFO: `/v3/groups` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.268687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c07ab0e-a623-43cd-9885-0062d978a82e None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:25.270111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c07ab0e-a623-43cd-9885-0062d978a82e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:25.287451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 156/325] 10.222.0.25 () {60 vars in 1134 bytes} [Sun Oct 2 00:41:25 2022] POST /identity/v3/groups => generated 226 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:41:25.342382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 170/326] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:25.383679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c07ab0e-a623-43cd-9885-0062d978a82e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:25.476525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:25.495812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c07ab0e-a623-43cd-9885-0062d978a82e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:25.528628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b4489d-5f6f-4154-92f8-c6676f15350f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:25.529257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b4489d-5f6f-4154-92f8-c6676f15350f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.529525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b4489d-5f6f-4154-92f8-c6676f15350f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.529802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b4489d-5f6f-4154-92f8-c6676f15350f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.539801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b4489d-5f6f-4154-92f8-c6676f15350f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-99b4489d-5f6f-4154-92f8-c6676f15350f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:25.545717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:25.546695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:25.547479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 157/327] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:25.554103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d81d9-a97a-4363-8a0d-112cde389330 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:25.602591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b4489d-5f6f-4154-92f8-c6676f15350f None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:25.659335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:25.671512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9d81d9-a97a-4363-8a0d-112cde389330 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:25.710661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c85712-59b5-443f-8373-e8f9879960c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:25.711270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c85712-59b5-443f-8373-e8f9879960c4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.711549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c85712-59b5-443f-8373-e8f9879960c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.711821 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c85712-59b5-443f-8373-e8f9879960c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.712482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c85712-59b5-443f-8373-e8f9879960c4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:25.712856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c85712-59b5-443f-8373-e8f9879960c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:25.714272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c85712-59b5-443f-8373-e8f9879960c4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:25.723765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 171/328] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:25.725701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b4489d-5f6f-4154-92f8-c6676f15350f None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:25.729468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c85712-59b5-443f-8373-e8f9879960c4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:25.775497 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f15bf1-b7e3-4d78-855b-3acb3cabe8fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:25.776064 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f15bf1-b7e3-4d78-855b-3acb3cabe8fa None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.776337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f15bf1-b7e3-4d78-855b-3acb3cabe8fa None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.776630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f15bf1-b7e3-4d78-855b-3acb3cabe8fa None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.793470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f15bf1-b7e3-4d78-855b-3acb3cabe8fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-d8f15bf1-b7e3-4d78-855b-3acb3cabe8fa None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:25.806149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:25.808113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:25.809455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:25.810554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 158/329] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:25.815624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f15bf1-b7e3-4d78-855b-3acb3cabe8fa None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:25.826405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:25.872678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c85712-59b5-443f-8373-e8f9879960c4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:25.920858 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695f8d9a-f9a5-4178-b15e-257373b5548f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:25.921498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695f8d9a-f9a5-4178-b15e-257373b5548f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.921766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695f8d9a-f9a5-4178-b15e-257373b5548f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.922033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695f8d9a-f9a5-4178-b15e-257373b5548f None admin] PATH_INFO: `/v3/users/c09c126a57144a5aa89bf76276b2f0a8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.923218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695f8d9a-f9a5-4178-b15e-257373b5548f None admin] RBAC: Authorizing `identity:get_user(user_id=c09c126a57144a5aa89bf76276b2f0a8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:25.924647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695f8d9a-f9a5-4178-b15e-257373b5548f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:25.927043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 172/330] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:25 2022] GET /identity/v3/users/c09c126a57144a5aa89bf76276b2f0a8 => generated 293 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:25.933645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f15bf1-b7e3-4d78-855b-3acb3cabe8fa None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:25.934392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695f8d9a-f9a5-4178-b15e-257373b5548f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:25.961126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:25.980827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21fd1ec-ba2f-4949-b51a-2732d0f47eb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:25.982029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21fd1ec-ba2f-4949-b51a-2732d0f47eb1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:25.982337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21fd1ec-ba2f-4949-b51a-2732d0f47eb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:25.982613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21fd1ec-ba2f-4949-b51a-2732d0f47eb1 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:25.983743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21fd1ec-ba2f-4949-b51a-2732d0f47eb1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:25.984060 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21fd1ec-ba2f-4949-b51a-2732d0f47eb1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:25.985673 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21fd1ec-ba2f-4949-b51a-2732d0f47eb1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:25.998219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 159/331] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:26.003191 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21fd1ec-ba2f-4949-b51a-2732d0f47eb1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:26.065134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695f8d9a-f9a5-4178-b15e-257373b5548f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:26.090909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:26.111218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8362585-1241-440a-98a9-2a521a2b7ef2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:26.112316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8362585-1241-440a-98a9-2a521a2b7ef2 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:26.112783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8362585-1241-440a-98a9-2a521a2b7ef2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:26.113242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8362585-1241-440a-98a9-2a521a2b7ef2 None admin] PATH_INFO: `/v3/system/users/c09c126a57144a5aa89bf76276b2f0a8/roles/680bd58399004020984edb53af05d3f8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:26.115021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8362585-1241-440a-98a9-2a521a2b7ef2 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=c09c126a57144a5aa89bf76276b2f0a8, role_id=680bd58399004020984edb53af05d3f8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:26.116606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8362585-1241-440a-98a9-2a521a2b7ef2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:26.130537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 173/332] 10.222.0.25 () {58 vars in 1334 bytes} [Sun Oct 2 00:41:25 2022] PUT /identity/v3/system/users/c09c126a57144a5aa89bf76276b2f0a8/roles/680bd58399004020984edb53af05d3f8 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:41:26.141309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21fd1ec-ba2f-4949-b51a-2732d0f47eb1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:26.196258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818aaee5-dd7c-4c91-940e-32edf3f74828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:26.196997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818aaee5-dd7c-4c91-940e-32edf3f74828 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:26.197867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818aaee5-dd7c-4c91-940e-32edf3f74828 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:26.198218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818aaee5-dd7c-4c91-940e-32edf3f74828 None admin] PATH_INFO: `/v3/users/863367eeb7ab443ca8c269f2f5e8e58c` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:26.199500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818aaee5-dd7c-4c91-940e-32edf3f74828 None admin] RBAC: Authorizing `identity:get_user(user_id=863367eeb7ab443ca8c269f2f5e8e58c)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:26.201071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818aaee5-dd7c-4c91-940e-32edf3f74828 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:26.203333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 160/333] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:26 2022] GET /identity/v3/users/863367eeb7ab443ca8c269f2f5e8e58c => generated 293 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:26.209694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8362585-1241-440a-98a9-2a521a2b7ef2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:26.332452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:26.350537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8362585-1241-440a-98a9-2a521a2b7ef2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:26.391803 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150caee3-fa23-4447-bb71-1b3f52b5abe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:26.392411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150caee3-fa23-4447-bb71-1b3f52b5abe5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:26.392692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150caee3-fa23-4447-bb71-1b3f52b5abe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:26.392961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150caee3-fa23-4447-bb71-1b3f52b5abe5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:26.393695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150caee3-fa23-4447-bb71-1b3f52b5abe5 None admin] RBAC: Adding query filter params (role.id=e4fa1b2d8c404336a855b6142fbf25d4, user.id=863367eeb7ab443ca8c269f2f5e8e58c, scope.system=all) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:26.394003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150caee3-fa23-4447-bb71-1b3f52b5abe5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:26.395471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150caee3-fa23-4447-bb71-1b3f52b5abe5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:26.408367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 174/334] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Oct 2 00:41:26 2022] GET /identity/v3/role_assignments?user.id=863367eeb7ab443ca8c269f2f5e8e58c&scope.system=all&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 510 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:27.815551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d213f4ef-e0a5-4ed4-b179-ff8b19dc8610 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:27.816056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d213f4ef-e0a5-4ed4-b179-ff8b19dc8610 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:27.816159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d213f4ef-e0a5-4ed4-b179-ff8b19dc8610 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:27.817425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 161/335] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:27.824026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a1adc8-2c81-42e5-8a6d-20de57ce136e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:27.824422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a1adc8-2c81-42e5-8a6d-20de57ce136e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:27.824685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a1adc8-2c81-42e5-8a6d-20de57ce136e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:27.845901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b5a1adc8-2c81-42e5-8a6d-20de57ce136e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:27.847139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b5a1adc8-2c81-42e5-8a6d-20de57ce136e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:27.921477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 175/336] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:28.107586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b098b862-28d6-4537-b299-83fd448c0b14 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:28.107864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b098b862-28d6-4537-b299-83fd448c0b14 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:28.108061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b098b862-28d6-4537-b299-83fd448c0b14 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:28.109613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 162/337] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:28.115100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a396a793-394e-466e-aae0-a1caec9f57f9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:28.115382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a396a793-394e-466e-aae0-a1caec9f57f9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:28.115657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a396a793-394e-466e-aae0-a1caec9f57f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:28.143574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a396a793-394e-466e-aae0-a1caec9f57f9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:28.144754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a396a793-394e-466e-aae0-a1caec9f57f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:28.220535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 176/338] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:28.507917 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b098b862-28d6-4537-b299-83fd448c0b14 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:28.623775 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b098b862-28d6-4537-b299-83fd448c0b14 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:28.669642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d40e97-9c21-453a-8b5b-071e991e6554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:28.670339 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d40e97-9c21-453a-8b5b-071e991e6554 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:28.670624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d40e97-9c21-453a-8b5b-071e991e6554 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:28.670895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d40e97-9c21-453a-8b5b-071e991e6554 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:28.673859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:28.688136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d40e97-9c21-453a-8b5b-071e991e6554 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-70d40e97-9c21-453a-8b5b-071e991e6554 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:28.693101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:28.693969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:28.695264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 163/339] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:28.703166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a396a793-394e-466e-aae0-a1caec9f57f9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:28.711174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:28.744139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d40e97-9c21-453a-8b5b-071e991e6554 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:28.890567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a396a793-394e-466e-aae0-a1caec9f57f9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:28.912457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d40e97-9c21-453a-8b5b-071e991e6554 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:28.954498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7dfab4-4f21-4272-9b8a-96e916344152 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:28.955142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7dfab4-4f21-4272-9b8a-96e916344152 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:28.955417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7dfab4-4f21-4272-9b8a-96e916344152 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:28.955694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7dfab4-4f21-4272-9b8a-96e916344152 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:28.956383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7dfab4-4f21-4272-9b8a-96e916344152 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:28.956820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7dfab4-4f21-4272-9b8a-96e916344152 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:28.958375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7dfab4-4f21-4272-9b8a-96e916344152 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:28.974925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 177/340] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:28.976962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62eb66ef-c9b8-4bfd-94c0-948df4323280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:28.977605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eb66ef-c9b8-4bfd-94c0-948df4323280 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:28.977881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eb66ef-c9b8-4bfd-94c0-948df4323280 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:28.978270 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eb66ef-c9b8-4bfd-94c0-948df4323280 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:28.982216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7dfab4-4f21-4272-9b8a-96e916344152 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:28.996395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62eb66ef-c9b8-4bfd-94c0-948df4323280 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-62eb66ef-c9b8-4bfd-94c0-948df4323280 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:29.001812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:29.002723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:29.003500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:29.007319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 164/341] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:29.013256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62eb66ef-c9b8-4bfd-94c0-948df4323280 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:29.118430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:29.123547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:29.189662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7dfab4-4f21-4272-9b8a-96e916344152 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:29.207550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62eb66ef-c9b8-4bfd-94c0-948df4323280 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:29.257021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d4097c-669d-42a6-9a5d-dea3b6225afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:29.257628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d4097c-669d-42a6-9a5d-dea3b6225afc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:29.257903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d4097c-669d-42a6-9a5d-dea3b6225afc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:29.258198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d4097c-669d-42a6-9a5d-dea3b6225afc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:29.280283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dfffc9-4de9-4fbf-98ff-ead06814a6ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:29.280900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dfffc9-4de9-4fbf-98ff-ead06814a6ed None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:29.281178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dfffc9-4de9-4fbf-98ff-ead06814a6ed None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:29.281452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dfffc9-4de9-4fbf-98ff-ead06814a6ed None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:29.282147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dfffc9-4de9-4fbf-98ff-ead06814a6ed None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:29.282457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dfffc9-4de9-4fbf-98ff-ead06814a6ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:29.283931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dfffc9-4de9-4fbf-98ff-ead06814a6ed None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:29.304154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 165/342] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:29.305494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a3b8c0-2225-4e68-996f-218d563124d5 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:29.306053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a3b8c0-2225-4e68-996f-218d563124d5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:29.306386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a3b8c0-2225-4e68-996f-218d563124d5 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:29.307664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 166/343] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:29.308362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a3b8c0-2225-4e68-996f-218d563124d5 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:29.397657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d4097c-669d-42a6-9a5d-dea3b6225afc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:29.399186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d4097c-669d-42a6-9a5d-dea3b6225afc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:29.487933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a3b8c0-2225-4e68-996f-218d563124d5 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:29.488287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:29.527529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:29.528434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 178/344] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 546 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:29.529780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a6c55b-e87e-49ed-b4ee-6e2d051fd3a5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:29.530083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a6c55b-e87e-49ed-b4ee-6e2d051fd3a5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:29.530789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a6c55b-e87e-49ed-b4ee-6e2d051fd3a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:29.550021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-97a6c55b-e87e-49ed-b4ee-6e2d051fd3a5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:29.551365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-97a6c55b-e87e-49ed-b4ee-6e2d051fd3a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:29.561610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced11e0a-42ea-4035-9a06-7994d465c2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:29.562653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced11e0a-42ea-4035-9a06-7994d465c2c4 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:29.563227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced11e0a-42ea-4035-9a06-7994d465c2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:29.563706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced11e0a-42ea-4035-9a06-7994d465c2c4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:29.626653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 179/345] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:29.627312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a6c55b-e87e-49ed-b4ee-6e2d051fd3a5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:29.653159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced11e0a-42ea-4035-9a06-7994d465c2c4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:29.655176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced11e0a-42ea-4035-9a06-7994d465c2c4 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:29.727411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:29.733399 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a6c55b-e87e-49ed-b4ee-6e2d051fd3a5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:29.766333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 167/346] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:29.768103 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e81483-1eba-4552-a6a0-c75740a7cb41 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:29.768580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e81483-1eba-4552-a6a0-c75740a7cb41 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:29.769039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e81483-1eba-4552-a6a0-c75740a7cb41 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:29.770644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 168/347] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:29.773177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf641e3-f25c-4824-be15-1d873ed6f0de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:29.773309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf641e3-f25c-4824-be15-1d873ed6f0de None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:29.773706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf641e3-f25c-4824-be15-1d873ed6f0de None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:29.774048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf641e3-f25c-4824-be15-1d873ed6f0de None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:29.774641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e81483-1eba-4552-a6a0-c75740a7cb41 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:29.778083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf641e3-f25c-4824-be15-1d873ed6f0de None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:29.778083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf641e3-f25c-4824-be15-1d873ed6f0de None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:29.779909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 180/348] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:29 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:29.781143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b611ca2e-a388-4ffa-95ce-42d575f59a56 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:29.781495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b611ca2e-a388-4ffa-95ce-42d575f59a56 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:29.781861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b611ca2e-a388-4ffa-95ce-42d575f59a56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:29.802629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b611ca2e-a388-4ffa-95ce-42d575f59a56 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:29.803269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:29.804064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b611ca2e-a388-4ffa-95ce-42d575f59a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:29.887868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 181/349] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:29.887868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b611ca2e-a388-4ffa-95ce-42d575f59a56 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:29.942747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e81483-1eba-4552-a6a0-c75740a7cb41 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:29.995884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:30.009038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b611ca2e-a388-4ffa-95ce-42d575f59a56 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:30.038818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09705b59-465b-4565-aefc-9246469ca2c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:30.039470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09705b59-465b-4565-aefc-9246469ca2c6 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:30.039749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09705b59-465b-4565-aefc-9246469ca2c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:30.040019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09705b59-465b-4565-aefc-9246469ca2c6 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:30.041724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09705b59-465b-4565-aefc-9246469ca2c6 None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:30.043195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09705b59-465b-4565-aefc-9246469ca2c6 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:30.045820 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 169/350] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:29 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:30.047030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3ef6f4-ab1d-4159-b758-c1d1648cad79 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:30.047306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3ef6f4-ab1d-4159-b758-c1d1648cad79 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:30.047576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3ef6f4-ab1d-4159-b758-c1d1648cad79 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:30.048775 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 170/351] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:30 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:30.053621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3ef6f4-ab1d-4159-b758-c1d1648cad79 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:30.066961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7c0424-e555-4445-bb19-fd85c7aa028b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:30.067231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7c0424-e555-4445-bb19-fd85c7aa028b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:30.067510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7c0424-e555-4445-bb19-fd85c7aa028b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:30.067884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7c0424-e555-4445-bb19-fd85c7aa028b None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:30.079586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7c0424-e555-4445-bb19-fd85c7aa028b None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:30.081048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7c0424-e555-4445-bb19-fd85c7aa028b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:30.083465 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 182/352] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:29 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:30.084436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c8ef5-8288-4764-a8ea-5b449cee4ad6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:30.084716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c8ef5-8288-4764-a8ea-5b449cee4ad6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:30.084990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c8ef5-8288-4764-a8ea-5b449cee4ad6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:30.121986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7f3c8ef5-8288-4764-a8ea-5b449cee4ad6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:30.124026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7f3c8ef5-8288-4764-a8ea-5b449cee4ad6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:30.179522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:30.207293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3ef6f4-ab1d-4159-b758-c1d1648cad79 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:30.210378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 183/353] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:30.211116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c8ef5-8288-4764-a8ea-5b449cee4ad6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:30.279572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c8d30-34a2-4f4d-b1c4-7958db4fdc1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:30.280213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c8d30-34a2-4f4d-b1c4-7958db4fdc1e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:30.280585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c8d30-34a2-4f4d-b1c4-7958db4fdc1e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:30.280855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c8d30-34a2-4f4d-b1c4-7958db4fdc1e None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:30.282118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410c8d30-34a2-4f4d-b1c4-7958db4fdc1e None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:30.283569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410c8d30-34a2-4f4d-b1c4-7958db4fdc1e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:30.285294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 171/354] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:30 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:30.287348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c8d30-34a2-4f4d-b1c4-7958db4fdc1e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:30.342209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:30.357776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c8ef5-8288-4764-a8ea-5b449cee4ad6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:30.394564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abeb7627-b859-4134-a6df-9f33f6433032 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:30.395318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeb7627-b859-4134-a6df-9f33f6433032 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:30.395694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeb7627-b859-4134-a6df-9f33f6433032 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:30.396056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeb7627-b859-4134-a6df-9f33f6433032 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:30.397105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abeb7627-b859-4134-a6df-9f33f6433032 None admin] RBAC: Adding query filter params (group.id=0e21a7774468486e9c4b62328a0325f2, role.id=e4fa1b2d8c404336a855b6142fbf25d4, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:30.397367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abeb7627-b859-4134-a6df-9f33f6433032 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:30.399332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abeb7627-b859-4134-a6df-9f33f6433032 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:30.405987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 184/355] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 2 00:41:30 2022] GET /identity/v3/role_assignments?group.id=0e21a7774468486e9c4b62328a0325f2&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 260 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:30.407023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abeb7627-b859-4134-a6df-9f33f6433032 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:30.458988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c8d30-34a2-4f4d-b1c4-7958db4fdc1e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:30.535136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976d1ddd-84a2-4058-a4aa-a96cc3a9da32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:30.535788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976d1ddd-84a2-4058-a4aa-a96cc3a9da32 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:30.536065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976d1ddd-84a2-4058-a4aa-a96cc3a9da32 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:30.536340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976d1ddd-84a2-4058-a4aa-a96cc3a9da32 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:30.555988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976d1ddd-84a2-4058-a4aa-a96cc3a9da32 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-976d1ddd-84a2-4058-a4aa-a96cc3a9da32 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:30.560975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:30.561954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 172/356] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:30.563147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976d1ddd-84a2-4058-a4aa-a96cc3a9da32 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:30.583608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:30.596744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abeb7627-b859-4134-a6df-9f33f6433032 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:30.648825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a48763a-318a-48dd-b369-59d268990073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:30.649439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48763a-318a-48dd-b369-59d268990073 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:30.649768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48763a-318a-48dd-b369-59d268990073 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:30.650134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48763a-318a-48dd-b369-59d268990073 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:30.650989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a48763a-318a-48dd-b369-59d268990073 None admin] RBAC: Adding query filter params (group.id=0e21a7774468486e9c4b62328a0325f2, role.id=5e19a16e5bfd4c94b1c34ae717ccfc42, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:30.651218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a48763a-318a-48dd-b369-59d268990073 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:30.652717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a48763a-318a-48dd-b369-59d268990073 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:30.660539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 185/357] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 2 00:41:30 2022] GET /identity/v3/role_assignments?group.id=0e21a7774468486e9c4b62328a0325f2&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 260 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:30.661460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a48763a-318a-48dd-b369-59d268990073 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:30.702944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:30.732217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976d1ddd-84a2-4058-a4aa-a96cc3a9da32 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:30.807473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b4cb95-2cf2-4d67-87c1-4c78b2399b9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:30.808043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b4cb95-2cf2-4d67-87c1-4c78b2399b9a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:30.808319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b4cb95-2cf2-4d67-87c1-4c78b2399b9a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:30.808588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b4cb95-2cf2-4d67-87c1-4c78b2399b9a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:30.832103 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b4cb95-2cf2-4d67-87c1-4c78b2399b9a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-89b4cb95-2cf2-4d67-87c1-4c78b2399b9a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:30.839716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:30.840741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:30.841582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 173/358] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:30 2022] GET /identity/v3/roles/reader => generated 84 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:30.842294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b4cb95-2cf2-4d67-87c1-4c78b2399b9a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:30.873746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a48763a-318a-48dd-b369-59d268990073 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:30.918880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4d62a9-b623-440b-98ef-553306937ea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:30.919581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4d62a9-b623-440b-98ef-553306937ea2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:30.919857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4d62a9-b623-440b-98ef-553306937ea2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:30.920125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4d62a9-b623-440b-98ef-553306937ea2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:30.932414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4d62a9-b623-440b-98ef-553306937ea2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bc4d62a9-b623-440b-98ef-553306937ea2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:30.937897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:30.938837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:30.939642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 186/359] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:30.940320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4d62a9-b623-440b-98ef-553306937ea2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:30.964584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:31.003104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b4cb95-2cf2-4d67-87c1-4c78b2399b9a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:31.054445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e903689-db78-4d96-b043-77f1d4734579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:31.055048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e903689-db78-4d96-b043-77f1d4734579 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:31.055320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e903689-db78-4d96-b043-77f1d4734579 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:31.055585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e903689-db78-4d96-b043-77f1d4734579 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:31.056349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e903689-db78-4d96-b043-77f1d4734579 None admin] RBAC: Adding query filter params (name=member) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:31.056650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e903689-db78-4d96-b043-77f1d4734579 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:31.058728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e903689-db78-4d96-b043-77f1d4734579 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:31.061071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:31.069692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 174/360] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:31.073935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e903689-db78-4d96-b043-77f1d4734579 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:31.095336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4d62a9-b623-440b-98ef-553306937ea2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:31.136524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecba9fcc-f4e8-4e82-9800-90b96d3ab3fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:31.137456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba9fcc-f4e8-4e82-9800-90b96d3ab3fa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:31.137635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba9fcc-f4e8-4e82-9800-90b96d3ab3fa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:31.137867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba9fcc-f4e8-4e82-9800-90b96d3ab3fa None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:31.138694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecba9fcc-f4e8-4e82-9800-90b96d3ab3fa None admin] RBAC: Adding query filter params (name=reader) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:31.138991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecba9fcc-f4e8-4e82-9800-90b96d3ab3fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:31.140697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecba9fcc-f4e8-4e82-9800-90b96d3ab3fa None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:31.153677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 187/361] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:30 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:31.155676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dabd2f3-1f1e-4962-baaa-b64079eda2f1 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:31.155676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dabd2f3-1f1e-4962-baaa-b64079eda2f1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:31.155676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dabd2f3-1f1e-4962-baaa-b64079eda2f1 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:31.157669 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 188/362] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:31.158818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dabd2f3-1f1e-4962-baaa-b64079eda2f1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:31.206901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e903689-db78-4d96-b043-77f1d4734579 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:31.248772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ab83e6-9b1d-44af-bef5-f1d061e77ac9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:31.249762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ab83e6-9b1d-44af-bef5-f1d061e77ac9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:31.252323 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ab83e6-9b1d-44af-bef5-f1d061e77ac9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:31.252922 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ab83e6-9b1d-44af-bef5-f1d061e77ac9 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:31.253761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ab83e6-9b1d-44af-bef5-f1d061e77ac9 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:31.254240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ab83e6-9b1d-44af-bef5-f1d061e77ac9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:31.258457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ab83e6-9b1d-44af-bef5-f1d061e77ac9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:31.262340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:31.272678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 175/363] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:31.273873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ab83e6-9b1d-44af-bef5-f1d061e77ac9 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:31.291905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dabd2f3-1f1e-4962-baaa-b64079eda2f1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:31.342767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94080e4-48ad-4942-9afa-d2b04770f1bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:31.343745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94080e4-48ad-4942-9afa-d2b04770f1bc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:31.344339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94080e4-48ad-4942-9afa-d2b04770f1bc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:31.344923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94080e4-48ad-4942-9afa-d2b04770f1bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:31.357319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:31.400580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ab83e6-9b1d-44af-bef5-f1d061e77ac9 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:31.441514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36d81dc-998d-4c90-a3a5-71afc84a001a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:31.442181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36d81dc-998d-4c90-a3a5-71afc84a001a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:31.442466 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36d81dc-998d-4c90-a3a5-71afc84a001a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:31.442736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36d81dc-998d-4c90-a3a5-71afc84a001a None admin] PATH_INFO: `/v3/users/c09c126a57144a5aa89bf76276b2f0a8` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:31.443672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94080e4-48ad-4942-9afa-d2b04770f1bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:31.443890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36d81dc-998d-4c90-a3a5-71afc84a001a None admin] RBAC: Authorizing `identity:get_user(user_id=c09c126a57144a5aa89bf76276b2f0a8)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:31.445276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36d81dc-998d-4c90-a3a5-71afc84a001a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:31.445894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94080e4-48ad-4942-9afa-d2b04770f1bc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:31.447628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 176/364] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:41:31 2022] GET /identity/v3/users/c09c126a57144a5aa89bf76276b2f0a8 => generated 293 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:31.453143 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a971fb0b-8dec-47d6-8ae5-e9137c5d386c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:31.453143 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a971fb0b-8dec-47d6-8ae5-e9137c5d386c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:31.453143 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a971fb0b-8dec-47d6-8ae5-e9137c5d386c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:31.472335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a971fb0b-8dec-47d6-8ae5-e9137c5d386c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:31.473551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a971fb0b-8dec-47d6-8ae5-e9137c5d386c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:31.548940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:31.557833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 177/365] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:31.558526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a971fb0b-8dec-47d6-8ae5-e9137c5d386c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:31.571577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 189/366] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:31.574290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94080e4-48ad-4942-9afa-d2b04770f1bc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:31.614912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:31.708402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94080e4-48ad-4942-9afa-d2b04770f1bc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:31.724919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a971fb0b-8dec-47d6-8ae5-e9137c5d386c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:31.767609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922ca9a2-2d8f-42f1-83f5-fd3fa5c247bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:31.770194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922ca9a2-2d8f-42f1-83f5-fd3fa5c247bf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:31.772268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922ca9a2-2d8f-42f1-83f5-fd3fa5c247bf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:31.772803 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922ca9a2-2d8f-42f1-83f5-fd3fa5c247bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:31.774636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922ca9a2-2d8f-42f1-83f5-fd3fa5c247bf None admin] RBAC: Adding query filter params (role.id=680bd58399004020984edb53af05d3f8, user.id=c09c126a57144a5aa89bf76276b2f0a8, scope.system=all) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:31.775966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922ca9a2-2d8f-42f1-83f5-fd3fa5c247bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:31.777708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922ca9a2-2d8f-42f1-83f5-fd3fa5c247bf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:31.794129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd71ef6-af11-4d5f-a905-04ae39e72223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:31.795088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd71ef6-af11-4d5f-a905-04ae39e72223 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:31.795584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd71ef6-af11-4d5f-a905-04ae39e72223 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:31.796060 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd71ef6-af11-4d5f-a905-04ae39e72223 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:31.802748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 190/367] 10.222.0.25 () {56 vars in 1308 bytes} [Sun Oct 2 00:41:31 2022] GET /identity/v3/role_assignments?user.id=c09c126a57144a5aa89bf76276b2f0a8&scope.system=all&role.id=680bd58399004020984edb53af05d3f8 => generated 510 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:31.812594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922ca9a2-2d8f-42f1-83f5-fd3fa5c247bf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:31.941755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:31.954563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd71ef6-af11-4d5f-a905-04ae39e72223 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:31.956294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd71ef6-af11-4d5f-a905-04ae39e72223 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:31.995145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922ca9a2-2d8f-42f1-83f5-fd3fa5c247bf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:32.008666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:32.040663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2a72a-d34f-4fa3-aac8-3a8edd5cf602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:32.041623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2a72a-d34f-4fa3-aac8-3a8edd5cf602 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:32.042177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2a72a-d34f-4fa3-aac8-3a8edd5cf602 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:32.042627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2a72a-d34f-4fa3-aac8-3a8edd5cf602 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:32.043849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db2a72a-d34f-4fa3-aac8-3a8edd5cf602 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:32.045333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db2a72a-d34f-4fa3-aac8-3a8edd5cf602 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:32.047079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 191/368] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:31 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:32.048094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2a72a-d34f-4fa3-aac8-3a8edd5cf602 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:32.170744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2a72a-d34f-4fa3-aac8-3a8edd5cf602 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:32.190829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 178/369] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 632 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:32.191827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd71ef6-af11-4d5f-a905-04ae39e72223 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:32.218401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49155e7-ae2a-4d6a-ba95-45f25b5e09d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:32.219285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49155e7-ae2a-4d6a-ba95-45f25b5e09d9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:32.219570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49155e7-ae2a-4d6a-ba95-45f25b5e09d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:32.219845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49155e7-ae2a-4d6a-ba95-45f25b5e09d9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:32.235824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49155e7-ae2a-4d6a-ba95-45f25b5e09d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c49155e7-ae2a-4d6a-ba95-45f25b5e09d9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:32.241742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:32.242758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:32.244227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 192/370] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:41:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:32.244932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49155e7-ae2a-4d6a-ba95-45f25b5e09d9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:32.275812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:32.344611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:32.351654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd71ef6-af11-4d5f-a905-04ae39e72223 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:32.394729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49155e7-ae2a-4d6a-ba95-45f25b5e09d9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:32.396035 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff543fe-0b31-492e-89e0-cf79d44f1924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:32.396670 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff543fe-0b31-492e-89e0-cf79d44f1924 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:32.396947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff543fe-0b31-492e-89e0-cf79d44f1924 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:32.398142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff543fe-0b31-492e-89e0-cf79d44f1924 None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:32.398590 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff543fe-0b31-492e-89e0-cf79d44f1924 None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:32.401017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff543fe-0b31-492e-89e0-cf79d44f1924 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:32.403000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 179/371] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:32 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:32.403880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff543fe-0b31-492e-89e0-cf79d44f1924 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:32.438794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b988ef81-462a-46c5-8684-c6b3efa3f392 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:32.439854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b988ef81-462a-46c5-8684-c6b3efa3f392 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:32.440352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b988ef81-462a-46c5-8684-c6b3efa3f392 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:32.440828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b988ef81-462a-46c5-8684-c6b3efa3f392 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:32.442336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b988ef81-462a-46c5-8684-c6b3efa3f392 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:32.444003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b988ef81-462a-46c5-8684-c6b3efa3f392 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:32.450303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 193/372] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:32 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:32.451448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b988ef81-462a-46c5-8684-c6b3efa3f392 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:32.527502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff543fe-0b31-492e-89e0-cf79d44f1924 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:32.564011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:32.576729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b988ef81-462a-46c5-8684-c6b3efa3f392 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:32.585099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850839b7-a538-454f-a4c3-0f85475a7c9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:32.586043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850839b7-a538-454f-a4c3-0f85475a7c9b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:32.586907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850839b7-a538-454f-a4c3-0f85475a7c9b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:32.587715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850839b7-a538-454f-a4c3-0f85475a7c9b None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:32.588399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850839b7-a538-454f-a4c3-0f85475a7c9b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:32.588703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850839b7-a538-454f-a4c3-0f85475a7c9b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:32.590212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850839b7-a538-454f-a4c3-0f85475a7c9b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:32.600429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 180/373] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:32.601337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850839b7-a538-454f-a4c3-0f85475a7c9b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:32.621719 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bf90c5-5534-4894-9c9f-13db8a2778c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:32.623228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bf90c5-5534-4894-9c9f-13db8a2778c3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:32.623974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bf90c5-5534-4894-9c9f-13db8a2778c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:32.624462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bf90c5-5534-4894-9c9f-13db8a2778c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:32.625699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bf90c5-5534-4894-9c9f-13db8a2778c3 None admin] RBAC: Adding query filter params (group.id=0e21a7774468486e9c4b62328a0325f2, role.id=e4fa1b2d8c404336a855b6142fbf25d4, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:32.626470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bf90c5-5534-4894-9c9f-13db8a2778c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:32.627911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bf90c5-5534-4894-9c9f-13db8a2778c3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:32.637879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 194/374] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 2 00:41:32 2022] GET /identity/v3/role_assignments?group.id=0e21a7774468486e9c4b62328a0325f2&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 260 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:32.638765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:32.640246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bf90c5-5534-4894-9c9f-13db8a2778c3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:32.751817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850839b7-a538-454f-a4c3-0f85475a7c9b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:32.783259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bf90c5-5534-4894-9c9f-13db8a2778c3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:32.824882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7eb030e-6e13-4839-8736-308f0235dc2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:32.826254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7eb030e-6e13-4839-8736-308f0235dc2e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:32.827729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a40669e-5717-469a-814a-f71da6ad7966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:32.828272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7eb030e-6e13-4839-8736-308f0235dc2e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:32.829138 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a40669e-5717-469a-814a-f71da6ad7966 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:32.829630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7eb030e-6e13-4839-8736-308f0235dc2e None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:32.830405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a40669e-5717-469a-814a-f71da6ad7966 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:32.831344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a40669e-5717-469a-814a-f71da6ad7966 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:32.832203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7eb030e-6e13-4839-8736-308f0235dc2e None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:32.835113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7eb030e-6e13-4839-8736-308f0235dc2e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:32.840021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 181/375] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:32 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:32.846077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7eb030e-6e13-4839-8736-308f0235dc2e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:32.896790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:32.951180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a40669e-5717-469a-814a-f71da6ad7966 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:32.952733 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a40669e-5717-469a-814a-f71da6ad7966 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:32.984608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:32.995698 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7eb030e-6e13-4839-8736-308f0235dc2e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:33.044221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f3a3ca-6c44-4dac-bc39-c89549941271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:33.045458 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f3a3ca-6c44-4dac-bc39-c89549941271 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:33.046883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f3a3ca-6c44-4dac-bc39-c89549941271 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:33.047381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f3a3ca-6c44-4dac-bc39-c89549941271 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:33.048329 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f3a3ca-6c44-4dac-bc39-c89549941271 None admin] RBAC: Adding query filter params (group.id=0e21a7774468486e9c4b62328a0325f2, role.id=5e19a16e5bfd4c94b1c34ae717ccfc42, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:33.048846 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f3a3ca-6c44-4dac-bc39-c89549941271 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:33.050549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f3a3ca-6c44-4dac-bc39-c89549941271 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:33.059635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 182/376] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 2 00:41:32 2022] GET /identity/v3/role_assignments?group.id=0e21a7774468486e9c4b62328a0325f2&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 260 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:33.061368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969a5286-064e-4151-8e76-d836aaec6e3b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:33.061872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969a5286-064e-4151-8e76-d836aaec6e3b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:33.062399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969a5286-064e-4151-8e76-d836aaec6e3b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:33.064903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 183/377] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:33.071247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f211c2ef-ba93-4a19-b8a5-98ea88fc3b5f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:33.072489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f211c2ef-ba93-4a19-b8a5-98ea88fc3b5f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:33.072975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f211c2ef-ba93-4a19-b8a5-98ea88fc3b5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:33.084528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 195/378] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:33.091311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a40669e-5717-469a-814a-f71da6ad7966 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:33.094984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f211c2ef-ba93-4a19-b8a5-98ea88fc3b5f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:33.096518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f211c2ef-ba93-4a19-b8a5-98ea88fc3b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:33.203645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:33.204363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 184/379] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:33.255723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a40669e-5717-469a-814a-f71da6ad7966 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:33.298522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d23421-4e9e-4868-bf1b-474fed38cd8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:33.299090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d23421-4e9e-4868-bf1b-474fed38cd8e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:33.299363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d23421-4e9e-4868-bf1b-474fed38cd8e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:33.299634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d23421-4e9e-4868-bf1b-474fed38cd8e None admin] PATH_INFO: `/v3/projects/2ea30d4949414e4cafefc43b9507a1d0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:33.301167 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d23421-4e9e-4868-bf1b-474fed38cd8e None admin] RBAC: Authorizing `identity:get_project(project_id=2ea30d4949414e4cafefc43b9507a1d0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:33.302659 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d23421-4e9e-4868-bf1b-474fed38cd8e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:33.304301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 196/380] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:33 2022] GET /identity/v3/projects/2ea30d4949414e4cafefc43b9507a1d0 => generated 344 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:33.310157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f211c2ef-ba93-4a19-b8a5-98ea88fc3b5f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:33.460622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:33.480691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f211c2ef-ba93-4a19-b8a5-98ea88fc3b5f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:33.518838 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c643c2-9972-41e3-8f05-4b578b198c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:33.519478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c643c2-9972-41e3-8f05-4b578b198c01 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:33.519753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c643c2-9972-41e3-8f05-4b578b198c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:33.520015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c643c2-9972-41e3-8f05-4b578b198c01 None admin] PATH_INFO: `/v3/groups/ee3ab5fff6fd48a587a82854986f7865` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:33.527947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c643c2-9972-41e3-8f05-4b578b198c01 None admin] RBAC: Authorizing `identity:get_group(group_id=ee3ab5fff6fd48a587a82854986f7865)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:33.529397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c643c2-9972-41e3-8f05-4b578b198c01 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:33.531492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d23421-4e9e-4868-bf1b-474fed38cd8e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:33.531666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 185/381] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:33 2022] GET /identity/v3/groups/ee3ab5fff6fd48a587a82854986f7865 => generated 229 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:33.537087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c643c2-9972-41e3-8f05-4b578b198c01 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:33.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c643c2-9972-41e3-8f05-4b578b198c01 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:33.700764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d23421-4e9e-4868-bf1b-474fed38cd8e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:33.727639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:33.730439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e85671-0589-4b3b-a7ea-6a1aea528468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:33.731300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e85671-0589-4b3b-a7ea-6a1aea528468 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:33.731832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e85671-0589-4b3b-a7ea-6a1aea528468 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:33.732112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e85671-0589-4b3b-a7ea-6a1aea528468 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:33.732946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e85671-0589-4b3b-a7ea-6a1aea528468 None admin] RBAC: Adding query filter params (group.id=ee3ab5fff6fd48a587a82854986f7865, role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=2ea30d4949414e4cafefc43b9507a1d0) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:33.733354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e85671-0589-4b3b-a7ea-6a1aea528468 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:33.735244 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88e85671-0589-4b3b-a7ea-6a1aea528468 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:33.741073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6909bea3-5b82-4215-9922-7d0d6d2ff51c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:33.741984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6909bea3-5b82-4215-9922-7d0d6d2ff51c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:33.742482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6909bea3-5b82-4215-9922-7d0d6d2ff51c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:33.742583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 186/382] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 2 00:41:33 2022] GET /identity/v3/role_assignments?group.id=ee3ab5fff6fd48a587a82854986f7865&scope.project.id=2ea30d4949414e4cafefc43b9507a1d0&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 260 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:33.743043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6909bea3-5b82-4215-9922-7d0d6d2ff51c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:33.750091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7436051-65cb-4c86-b8b3-3cef8d53d514 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:33.750091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7436051-65cb-4c86-b8b3-3cef8d53d514 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:33.750091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7436051-65cb-4c86-b8b3-3cef8d53d514 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:33.750091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 187/383] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:33.753123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c209d136-a149-480e-a1bb-09c40b43cf3d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:33.753413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c209d136-a149-480e-a1bb-09c40b43cf3d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:33.753695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c209d136-a149-480e-a1bb-09c40b43cf3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:33.771108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6909bea3-5b82-4215-9922-7d0d6d2ff51c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:33.774275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-c209d136-a149-480e-a1bb-09c40b43cf3d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:33.775474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c209d136-a149-480e-a1bb-09c40b43cf3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6909bea3-5b82-4215-9922-7d0d6d2ff51c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:33.782319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:33.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:33.784066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:33.786877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 197/384] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:33.787706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:33.794916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6909bea3-5b82-4215-9922-7d0d6d2ff51c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:33.858724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 188/385] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:33.979142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6909bea3-5b82-4215-9922-7d0d6d2ff51c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:34.018179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fc0761-ddb2-404f-abf8-aa160f77c5c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:34.018792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fc0761-ddb2-404f-abf8-aa160f77c5c7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:34.019066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fc0761-ddb2-404f-abf8-aa160f77c5c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:34.019334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fc0761-ddb2-404f-abf8-aa160f77c5c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:34.020003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fc0761-ddb2-404f-abf8-aa160f77c5c7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:34.020302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fc0761-ddb2-404f-abf8-aa160f77c5c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:34.021661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fc0761-ddb2-404f-abf8-aa160f77c5c7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:34.037846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 198/386] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:34.047305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c209d136-a149-480e-a1bb-09c40b43cf3d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:34.186930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fc0761-ddb2-404f-abf8-aa160f77c5c7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:34.194126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c209d136-a149-480e-a1bb-09c40b43cf3d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:34.235028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef54d0c6-5970-42f1-bc03-2cc5b40287d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:34.236455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef54d0c6-5970-42f1-bc03-2cc5b40287d5 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:34.236888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef54d0c6-5970-42f1-bc03-2cc5b40287d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:34.237332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef54d0c6-5970-42f1-bc03-2cc5b40287d5 None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:34.238835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef54d0c6-5970-42f1-bc03-2cc5b40287d5 None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:34.240787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef54d0c6-5970-42f1-bc03-2cc5b40287d5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:34.243085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 189/387] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:34 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:34.248647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef54d0c6-5970-42f1-bc03-2cc5b40287d5 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:34.267609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:34.270553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:34.371549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fc0761-ddb2-404f-abf8-aa160f77c5c7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:34.416800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef54d0c6-5970-42f1-bc03-2cc5b40287d5 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:34.458094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5f6f60-fc74-4743-bf64-a9b2103e42d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:34.458094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f6f60-fc74-4743-bf64-a9b2103e42d9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:34.458094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f6f60-fc74-4743-bf64-a9b2103e42d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:34.458094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5f6f60-fc74-4743-bf64-a9b2103e42d9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:34.466315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8742280-f7cb-4e73-affa-8ac57207dddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:34.467647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8742280-f7cb-4e73-affa-8ac57207dddc None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:34.468119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8742280-f7cb-4e73-affa-8ac57207dddc None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:34.468571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8742280-f7cb-4e73-affa-8ac57207dddc None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:34.472600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8742280-f7cb-4e73-affa-8ac57207dddc None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:34.473196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5f6f60-fc74-4743-bf64-a9b2103e42d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:34.474461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8742280-f7cb-4e73-affa-8ac57207dddc None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:34.476601 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 190/388] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:34 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0a5f6f60-fc74-4743-bf64-a9b2103e42d9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:34.479098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:34.479983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:34.480875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:34.481480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 199/389] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:34.487627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5f6f60-fc74-4743-bf64-a9b2103e42d9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:34.488149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8742280-f7cb-4e73-affa-8ac57207dddc None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:34.592621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:34.620950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8742280-f7cb-4e73-affa-8ac57207dddc None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:34.687820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5f6f60-fc74-4743-bf64-a9b2103e42d9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:34.690372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e72330a-a75b-4274-b6c6-dc6e8a65d69d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:34.690984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e72330a-a75b-4274-b6c6-dc6e8a65d69d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:34.691261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e72330a-a75b-4274-b6c6-dc6e8a65d69d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:34.691531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e72330a-a75b-4274-b6c6-dc6e8a65d69d None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:34.692194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e72330a-a75b-4274-b6c6-dc6e8a65d69d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:34.692495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e72330a-a75b-4274-b6c6-dc6e8a65d69d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:34.693867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e72330a-a75b-4274-b6c6-dc6e8a65d69d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:34.702390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:34.711745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 191/390] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:34.717480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e72330a-a75b-4274-b6c6-dc6e8a65d69d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:34.747907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1bb494-f5c0-4bfc-9485-489d23380333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:34.748573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1bb494-f5c0-4bfc-9485-489d23380333 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:34.748844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1bb494-f5c0-4bfc-9485-489d23380333 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:34.749111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1bb494-f5c0-4bfc-9485-489d23380333 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/groups/0e21a7774468486e9c4b62328a0325f2/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:34.751834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1bb494-f5c0-4bfc-9485-489d23380333 None admin] RBAC: Authorizing `identity:create_grant(project_id=85be2a1b4ed54b91bfefe25b0945913e, group_id=0e21a7774468486e9c4b62328a0325f2, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:34.754013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1bb494-f5c0-4bfc-9485-489d23380333 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:34.772476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 200/391] 10.222.0.25 () {58 vars in 1442 bytes} [Sun Oct 2 00:41:34 2022] PUT /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/groups/0e21a7774468486e9c4b62328a0325f2/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:41:34.880076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e72330a-a75b-4274-b6c6-dc6e8a65d69d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:34.928451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319838d1-2651-4991-aaf5-2728cce4c3eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:34.929222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319838d1-2651-4991-aaf5-2728cce4c3eb None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:34.929508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319838d1-2651-4991-aaf5-2728cce4c3eb None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:34.929786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319838d1-2651-4991-aaf5-2728cce4c3eb None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:34.931657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319838d1-2651-4991-aaf5-2728cce4c3eb None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:34.933136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319838d1-2651-4991-aaf5-2728cce4c3eb None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:34.935302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 192/392] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:34 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:34.940529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1bb494-f5c0-4bfc-9485-489d23380333 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:35.079109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1bb494-f5c0-4bfc-9485-489d23380333 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:35.119617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f52433-9b7a-4ab0-94e4-138129bc9c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:35.120628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f52433-9b7a-4ab0-94e4-138129bc9c6a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:35.121113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f52433-9b7a-4ab0-94e4-138129bc9c6a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:35.121614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f52433-9b7a-4ab0-94e4-138129bc9c6a None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:35.123083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f52433-9b7a-4ab0-94e4-138129bc9c6a None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:35.124735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f52433-9b7a-4ab0-94e4-138129bc9c6a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:35.126623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 201/393] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:34 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:35.132612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319838d1-2651-4991-aaf5-2728cce4c3eb None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:35.159971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:35.268292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319838d1-2651-4991-aaf5-2728cce4c3eb None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:35.304216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc95887a-f3cb-4a09-9aeb-6056e9efffb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:35.304892 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc95887a-f3cb-4a09-9aeb-6056e9efffb5 None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:35.305176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc95887a-f3cb-4a09-9aeb-6056e9efffb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:35.305448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc95887a-f3cb-4a09-9aeb-6056e9efffb5 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/groups/0e21a7774468486e9c4b62328a0325f2/roles/5e19a16e5bfd4c94b1c34ae717ccfc42` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:35.307762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc95887a-f3cb-4a09-9aeb-6056e9efffb5 None admin] RBAC: Authorizing `identity:create_grant(project_id=85be2a1b4ed54b91bfefe25b0945913e, group_id=0e21a7774468486e9c4b62328a0325f2, role_id=5e19a16e5bfd4c94b1c34ae717ccfc42)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:35.309284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc95887a-f3cb-4a09-9aeb-6056e9efffb5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:35.325525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 193/394] 10.222.0.25 () {58 vars in 1442 bytes} [Sun Oct 2 00:41:35 2022] PUT /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e/groups/0e21a7774468486e9c4b62328a0325f2/roles/5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:41:35.607793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3c0ed7-afee-4f17-8378-61d3c5fa5a16 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:35.608289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3c0ed7-afee-4f17-8378-61d3c5fa5a16 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:35.608289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3c0ed7-afee-4f17-8378-61d3c5fa5a16 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:35.609837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 202/395] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:35.616028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b74d59-12c3-4acd-bf01-556d401878d2 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:35.616337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b74d59-12c3-4acd-bf01-556d401878d2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:35.616579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b74d59-12c3-4acd-bf01-556d401878d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:35.642000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-b5b74d59-12c3-4acd-bf01-556d401878d2 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:35.643251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b5b74d59-12c3-4acd-bf01-556d401878d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:35.720553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 194/396] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:35.958325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5010dbcc-18b6-4c06-9747-81e7a8049bdc None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:35.958667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5010dbcc-18b6-4c06-9747-81e7a8049bdc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:35.958837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5010dbcc-18b6-4c06-9747-81e7a8049bdc None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:35.960317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 203/397] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:35.965831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897718cd-0a94-4b90-ba50-9260c7a72070 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:35.966142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897718cd-0a94-4b90-ba50-9260c7a72070 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:35.966418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897718cd-0a94-4b90-ba50-9260c7a72070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:35.972864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:35.987856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-897718cd-0a94-4b90-ba50-9260c7a72070 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:35.989106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-897718cd-0a94-4b90-ba50-9260c7a72070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:36.020226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5010dbcc-18b6-4c06-9747-81e7a8049bdc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:36.073325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 195/398] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:36.091135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:36.188901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5010dbcc-18b6-4c06-9747-81e7a8049bdc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:36.230488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2055c5-9f0d-4b8a-87e5-272b6ce670aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:36.231454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2055c5-9f0d-4b8a-87e5-272b6ce670aa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:36.231929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2055c5-9f0d-4b8a-87e5-272b6ce670aa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:36.232393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2055c5-9f0d-4b8a-87e5-272b6ce670aa None admin] PATH_INFO: `/v3/roles/member` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:36.247489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2055c5-9f0d-4b8a-87e5-272b6ce670aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1a2055c5-9f0d-4b8a-87e5-272b6ce670aa None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:36.252657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:36.253518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:36.254360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:36.262128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 204/399] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:36.263821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897718cd-0a94-4b90-ba50-9260c7a72070 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:36.405627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897718cd-0a94-4b90-ba50-9260c7a72070 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:36.442080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:36.447155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a7a598-011b-4756-8a59-561f6c472e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:36.447856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a7a598-011b-4756-8a59-561f6c472e16 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:36.448134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a7a598-011b-4756-8a59-561f6c472e16 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:36.448811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a7a598-011b-4756-8a59-561f6c472e16 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:36.450103 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a7a598-011b-4756-8a59-561f6c472e16 None admin] RBAC: Adding query filter params (name=member) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:36.450424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a7a598-011b-4756-8a59-561f6c472e16 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:36.450922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2055c5-9f0d-4b8a-87e5-272b6ce670aa None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:36.451995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a7a598-011b-4756-8a59-561f6c472e16 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:36.463409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 196/400] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:36.474029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a7a598-011b-4756-8a59-561f6c472e16 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:36.638535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:36.663016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2055c5-9f0d-4b8a-87e5-272b6ce670aa None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:36.684595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a7a598-011b-4756-8a59-561f6c472e16 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:36.726668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce23f34-9f65-475a-9c84-579cf40892fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:36.728274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce23f34-9f65-475a-9c84-579cf40892fb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:36.728784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce23f34-9f65-475a-9c84-579cf40892fb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:36.729256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce23f34-9f65-475a-9c84-579cf40892fb None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:36.757595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce23f34-9f65-475a-9c84-579cf40892fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fce23f34-9f65-475a-9c84-579cf40892fb None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:36.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:36.765520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:36.766356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:36.770283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64b449a-1ff1-44dd-a8b1-4fdf62bea12a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:36.771190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64b449a-1ff1-44dd-a8b1-4fdf62bea12a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:36.771669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64b449a-1ff1-44dd-a8b1-4fdf62bea12a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:36.772151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64b449a-1ff1-44dd-a8b1-4fdf62bea12a None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:36.774110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 205/401] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:36.775950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64b449a-1ff1-44dd-a8b1-4fdf62bea12a None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:36.777697 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64b449a-1ff1-44dd-a8b1-4fdf62bea12a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:36.785295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce23f34-9f65-475a-9c84-579cf40892fb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:36.786010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 197/402] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:36 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:36.789538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64b449a-1ff1-44dd-a8b1-4fdf62bea12a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:36.883631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:36.941444 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64b449a-1ff1-44dd-a8b1-4fdf62bea12a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:36.947172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce23f34-9f65-475a-9c84-579cf40892fb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:36.976499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8838c9-d6eb-4671-a43d-3ed434b5acaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:36.977127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8838c9-d6eb-4671-a43d-3ed434b5acaf None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:36.977404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8838c9-d6eb-4671-a43d-3ed434b5acaf None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:36.977701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8838c9-d6eb-4671-a43d-3ed434b5acaf None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:36.978930 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8838c9-d6eb-4671-a43d-3ed434b5acaf None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:36.980485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8838c9-d6eb-4671-a43d-3ed434b5acaf None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:36.982609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 198/403] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:36 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:36.983754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345fe5f0-e4dd-4a57-ac87-d8d40b25fa33 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:36.984046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345fe5f0-e4dd-4a57-ac87-d8d40b25fa33 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:36.984293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345fe5f0-e4dd-4a57-ac87-d8d40b25fa33 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:36.989668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 199/404] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:36.989668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345fe5f0-e4dd-4a57-ac87-d8d40b25fa33 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:36.996511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9032c686-72d0-4f6a-8f1e-0c968083272f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:36.997982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9032c686-72d0-4f6a-8f1e-0c968083272f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:36.998485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9032c686-72d0-4f6a-8f1e-0c968083272f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:36.998966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9032c686-72d0-4f6a-8f1e-0c968083272f None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:37.000268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9032c686-72d0-4f6a-8f1e-0c968083272f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:37.001191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9032c686-72d0-4f6a-8f1e-0c968083272f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:37.003368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9032c686-72d0-4f6a-8f1e-0c968083272f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:37.020022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 206/405] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:37.020022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c195e5c8-3d8f-44aa-8955-e773f8d0b913 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:37.020022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c195e5c8-3d8f-44aa-8955-e773f8d0b913 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:37.020022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c195e5c8-3d8f-44aa-8955-e773f8d0b913 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:37.047618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c195e5c8-3d8f-44aa-8955-e773f8d0b913 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:37.049040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c195e5c8-3d8f-44aa-8955-e773f8d0b913 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:37.059896 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:37.114340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345fe5f0-e4dd-4a57-ac87-d8d40b25fa33 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:37.132593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 207/406] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:37.133459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c195e5c8-3d8f-44aa-8955-e773f8d0b913 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:37.143996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:37.151760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521aacbf-70a7-41cb-a305-9ee305df640e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:37.152611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521aacbf-70a7-41cb-a305-9ee305df640e None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:37.152953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521aacbf-70a7-41cb-a305-9ee305df640e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:37.153295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521aacbf-70a7-41cb-a305-9ee305df640e None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/groups/0e21a7774468486e9c4b62328a0325f2/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:37.155442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521aacbf-70a7-41cb-a305-9ee305df640e None admin] RBAC: Authorizing `identity:create_grant(project_id=74b104206ad84d6bbb9d5fd926b6f1aa, group_id=0e21a7774468486e9c4b62328a0325f2, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:37.157009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521aacbf-70a7-41cb-a305-9ee305df640e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:37.184506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 200/407] 10.222.0.25 () {58 vars in 1442 bytes} [Sun Oct 2 00:41:36 2022] PUT /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/groups/0e21a7774468486e9c4b62328a0325f2/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:41:37.246168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c195e5c8-3d8f-44aa-8955-e773f8d0b913 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:37.284183 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d281c46c-a3ad-446e-8b92-28cec750b1e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:37.284834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d281c46c-a3ad-446e-8b92-28cec750b1e2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:37.285107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d281c46c-a3ad-446e-8b92-28cec750b1e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:37.285376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d281c46c-a3ad-446e-8b92-28cec750b1e2 None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:37.286585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d281c46c-a3ad-446e-8b92-28cec750b1e2 None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:37.288096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d281c46c-a3ad-446e-8b92-28cec750b1e2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:37.290098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 208/408] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:37 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:37.299227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521aacbf-70a7-41cb-a305-9ee305df640e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:37.432770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521aacbf-70a7-41cb-a305-9ee305df640e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:37.451760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d281c46c-a3ad-446e-8b92-28cec750b1e2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:37.466137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68a2e26-fe2f-47bf-9855-561a3ea662d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:37.466773 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68a2e26-fe2f-47bf-9855-561a3ea662d5 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:37.467045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68a2e26-fe2f-47bf-9855-561a3ea662d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:37.467412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68a2e26-fe2f-47bf-9855-561a3ea662d5 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:37.468430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:37.469484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68a2e26-fe2f-47bf-9855-561a3ea662d5 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:37.471118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68a2e26-fe2f-47bf-9855-561a3ea662d5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:37.473326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 201/409] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:37 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:37.479492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68a2e26-fe2f-47bf-9855-561a3ea662d5 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:37.554509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:37.588478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68a2e26-fe2f-47bf-9855-561a3ea662d5 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:37.615461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d281c46c-a3ad-446e-8b92-28cec750b1e2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:37.624832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342a51af-8322-47c1-9616-6a4d16ff95eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:37.625489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342a51af-8322-47c1-9616-6a4d16ff95eb None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:37.625763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342a51af-8322-47c1-9616-6a4d16ff95eb None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:37.626032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342a51af-8322-47c1-9616-6a4d16ff95eb None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/groups/0e21a7774468486e9c4b62328a0325f2/roles/5e19a16e5bfd4c94b1c34ae717ccfc42` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:37.628166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342a51af-8322-47c1-9616-6a4d16ff95eb None admin] RBAC: Authorizing `identity:create_grant(project_id=74b104206ad84d6bbb9d5fd926b6f1aa, group_id=0e21a7774468486e9c4b62328a0325f2, role_id=5e19a16e5bfd4c94b1c34ae717ccfc42)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:37.629600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342a51af-8322-47c1-9616-6a4d16ff95eb None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:37.650138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 202/410] 10.222.0.25 () {58 vars in 1442 bytes} [Sun Oct 2 00:41:37 2022] PUT /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa/groups/0e21a7774468486e9c4b62328a0325f2/roles/5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:41:37.650138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fefe0ff-f1cd-433c-b25a-9afc845654ea None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:37.650138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fefe0ff-f1cd-433c-b25a-9afc845654ea None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:37.650138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fefe0ff-f1cd-433c-b25a-9afc845654ea None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:37.650138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 203/411] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:37.663890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49835767-6768-4d58-a452-3ab207f23e69 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:37.664469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49835767-6768-4d58-a452-3ab207f23e69 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:37.664880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49835767-6768-4d58-a452-3ab207f23e69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:37.698761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485bbed3-c19c-40b6-8286-71405ba12ad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:37.700036 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485bbed3-c19c-40b6-8286-71405ba12ad2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:37.700449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485bbed3-c19c-40b6-8286-71405ba12ad2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:37.700906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485bbed3-c19c-40b6-8286-71405ba12ad2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:37.705487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-49835767-6768-4d58-a452-3ab207f23e69 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:37.709880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-49835767-6768-4d58-a452-3ab207f23e69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:37.731867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485bbed3-c19c-40b6-8286-71405ba12ad2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-485bbed3-c19c-40b6-8286-71405ba12ad2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:37.743397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:37.744618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:37.745587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:37.748631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 209/412] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:41:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:37.754716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485bbed3-c19c-40b6-8286-71405ba12ad2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:37.849937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:37.864461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 204/413] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:37.885799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485bbed3-c19c-40b6-8286-71405ba12ad2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:37.924295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde3454-2811-4aa9-bc8b-432e9a46cf98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:37.925150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde3454-2811-4aa9-bc8b-432e9a46cf98 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:37.925560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde3454-2811-4aa9-bc8b-432e9a46cf98 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:37.925962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde3454-2811-4aa9-bc8b-432e9a46cf98 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:37.926769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddde3454-2811-4aa9-bc8b-432e9a46cf98 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:37.927204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddde3454-2811-4aa9-bc8b-432e9a46cf98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:37.928777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddde3454-2811-4aa9-bc8b-432e9a46cf98 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:37.939328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 210/414] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:37.944492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49835767-6768-4d58-a452-3ab207f23e69 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:37.996048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:38.075876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49835767-6768-4d58-a452-3ab207f23e69 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:38.124795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39ca7d2-3e88-4a91-8782-e88309711d47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:38.125367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39ca7d2-3e88-4a91-8782-e88309711d47 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:38.125642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39ca7d2-3e88-4a91-8782-e88309711d47 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:38.125911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39ca7d2-3e88-4a91-8782-e88309711d47 None admin] PATH_INFO: `/v3/groups/ee3ab5fff6fd48a587a82854986f7865` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:38.130995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39ca7d2-3e88-4a91-8782-e88309711d47 None admin] RBAC: Authorizing `identity:get_group(group_id=ee3ab5fff6fd48a587a82854986f7865)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:38.132492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39ca7d2-3e88-4a91-8782-e88309711d47 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:38.134615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 205/415] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:37 2022] GET /identity/v3/groups/ee3ab5fff6fd48a587a82854986f7865 => generated 229 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:38.140753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde3454-2811-4aa9-bc8b-432e9a46cf98 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:38.253898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde3454-2811-4aa9-bc8b-432e9a46cf98 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:38.278589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b80b8c-1618-44a4-b70c-d32d111b7063 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:38.279199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b80b8c-1618-44a4-b70c-d32d111b7063 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:38.279476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b80b8c-1618-44a4-b70c-d32d111b7063 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:38.281305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 206/416] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:38.287053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7da37a6-5776-4d81-bbe5-faec9bfbb7ec None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:38.287330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7da37a6-5776-4d81-bbe5-faec9bfbb7ec None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:38.287599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7da37a6-5776-4d81-bbe5-faec9bfbb7ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:38.297037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddc54a1-2feb-4c91-95e4-4697a698fbb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:38.297971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddc54a1-2feb-4c91-95e4-4697a698fbb5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:38.298500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddc54a1-2feb-4c91-95e4-4697a698fbb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:38.298986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddc54a1-2feb-4c91-95e4-4697a698fbb5 None admin] PATH_INFO: `/v3/projects/2ea30d4949414e4cafefc43b9507a1d0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:38.300338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddc54a1-2feb-4c91-95e4-4697a698fbb5 None admin] RBAC: Authorizing `identity:get_project(project_id=2ea30d4949414e4cafefc43b9507a1d0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:38.302008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eddc54a1-2feb-4c91-95e4-4697a698fbb5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:38.304197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 211/417] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:38 2022] GET /identity/v3/projects/2ea30d4949414e4cafefc43b9507a1d0 => generated 344 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:38.315103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddc54a1-2feb-4c91-95e4-4697a698fbb5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:38.316475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-b7da37a6-5776-4d81-bbe5-faec9bfbb7ec None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:38.318112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b7da37a6-5776-4d81-bbe5-faec9bfbb7ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:38.324472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:38.428944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 207/418] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:38.429740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7da37a6-5776-4d81-bbe5-faec9bfbb7ec None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:38.476306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddc54a1-2feb-4c91-95e4-4697a698fbb5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:38.480173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:38.559415 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb99df5-af97-44b1-9a3e-71adfbdc5896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:38.560084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb99df5-af97-44b1-9a3e-71adfbdc5896 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:38.560357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb99df5-af97-44b1-9a3e-71adfbdc5896 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:38.560624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb99df5-af97-44b1-9a3e-71adfbdc5896 None admin] PATH_INFO: `/v3/projects/2ea30d4949414e4cafefc43b9507a1d0/groups/ee3ab5fff6fd48a587a82854986f7865/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:38.563392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb99df5-af97-44b1-9a3e-71adfbdc5896 None admin] RBAC: Authorizing `identity:create_grant(project_id=2ea30d4949414e4cafefc43b9507a1d0, group_id=ee3ab5fff6fd48a587a82854986f7865, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:38.564836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb99df5-af97-44b1-9a3e-71adfbdc5896 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:38.571939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7da37a6-5776-4d81-bbe5-faec9bfbb7ec None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:38.590474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 212/419] 10.222.0.25 () {58 vars in 1442 bytes} [Sun Oct 2 00:41:38 2022] PUT /identity/v3/projects/2ea30d4949414e4cafefc43b9507a1d0/groups/ee3ab5fff6fd48a587a82854986f7865/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:41:38.640789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e107d018-7c11-4065-9bfb-0e0dc0089e51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:38.641707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e107d018-7c11-4065-9bfb-0e0dc0089e51 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:38.642075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e107d018-7c11-4065-9bfb-0e0dc0089e51 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:38.642349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e107d018-7c11-4065-9bfb-0e0dc0089e51 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:38.654946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e107d018-7c11-4065-9bfb-0e0dc0089e51 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-e107d018-7c11-4065-9bfb-0e0dc0089e51 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:38.660105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:38.661056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:38.662084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 208/420] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:38.668912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb99df5-af97-44b1-9a3e-71adfbdc5896 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:38.772183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e107d018-7c11-4065-9bfb-0e0dc0089e51 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:38.781502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:38.839471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb99df5-af97-44b1-9a3e-71adfbdc5896 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:38.884331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5f606b-05fc-4034-8816-d0074f646e87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:38.884969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5f606b-05fc-4034-8816-d0074f646e87 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:38.885241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5f606b-05fc-4034-8816-d0074f646e87 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:38.885507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5f606b-05fc-4034-8816-d0074f646e87 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:38.886523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5f606b-05fc-4034-8816-d0074f646e87 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:38.886850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5f606b-05fc-4034-8816-d0074f646e87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:38.889145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5f606b-05fc-4034-8816-d0074f646e87 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:38.898758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 213/421] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:38.904689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5f606b-05fc-4034-8816-d0074f646e87 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:38.944760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e107d018-7c11-4065-9bfb-0e0dc0089e51 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:38.958487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:39.016303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ea31a6-3845-4ec4-8f8b-d121df4b7e51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:39.017170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ea31a6-3845-4ec4-8f8b-d121df4b7e51 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:39.017589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ea31a6-3845-4ec4-8f8b-d121df4b7e51 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:39.018001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ea31a6-3845-4ec4-8f8b-d121df4b7e51 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:39.042093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ea31a6-3845-4ec4-8f8b-d121df4b7e51 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-65ea31a6-3845-4ec4-8f8b-d121df4b7e51 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:39.045076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:39.045966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:39.046782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:39.056032 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5f606b-05fc-4034-8816-d0074f646e87 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:39.063768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 209/422] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:39.064662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ea31a6-3845-4ec4-8f8b-d121df4b7e51 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:39.105950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:39.122357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ce20e5-435c-4e3f-a139-a43881b1104a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:39.122967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ce20e5-435c-4e3f-a139-a43881b1104a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:39.123243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ce20e5-435c-4e3f-a139-a43881b1104a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:39.123513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ce20e5-435c-4e3f-a139-a43881b1104a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:39.261862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ea31a6-3845-4ec4-8f8b-d121df4b7e51 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:39.263083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ce20e5-435c-4e3f-a139-a43881b1104a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:39.264557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ce20e5-435c-4e3f-a139-a43881b1104a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:39.299243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480a0544-04b3-4060-bfbb-56ce6234cada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:39.300165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480a0544-04b3-4060-bfbb-56ce6234cada None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:39.300587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480a0544-04b3-4060-bfbb-56ce6234cada None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:39.300865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480a0544-04b3-4060-bfbb-56ce6234cada None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:39.301664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480a0544-04b3-4060-bfbb-56ce6234cada None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:39.301968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480a0544-04b3-4060-bfbb-56ce6234cada None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:39.303509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480a0544-04b3-4060-bfbb-56ce6234cada None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:39.312482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 210/423] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:39 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:39.318898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480a0544-04b3-4060-bfbb-56ce6234cada None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:39.366436 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:39.426534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480a0544-04b3-4060-bfbb-56ce6234cada None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:39.473329 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7adcc4-d368-45e5-9ce2-63952a5e8248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:39.473948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7adcc4-d368-45e5-9ce2-63952a5e8248 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:39.474252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7adcc4-d368-45e5-9ce2-63952a5e8248 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:39.474527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7adcc4-d368-45e5-9ce2-63952a5e8248 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:39.540854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 214/424] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 636 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 02 00:41:39.541784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ce20e5-435c-4e3f-a139-a43881b1104a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:39.584486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:39.605228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7adcc4-d368-45e5-9ce2-63952a5e8248 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:39.606712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7adcc4-d368-45e5-9ce2-63952a5e8248 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:39.638359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:39.732606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 211/425] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 02 00:41:39.743256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7adcc4-d368-45e5-9ce2-63952a5e8248 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:39.743563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ce20e5-435c-4e3f-a139-a43881b1104a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:39.783779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7ad9e9-b3e7-47bc-94ee-9857166904e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:39.784395 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7ad9e9-b3e7-47bc-94ee-9857166904e8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:39.784670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7ad9e9-b3e7-47bc-94ee-9857166904e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:39.784942 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7ad9e9-b3e7-47bc-94ee-9857166904e8 None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:39.786652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7ad9e9-b3e7-47bc-94ee-9857166904e8 None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:39.788564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7ad9e9-b3e7-47bc-94ee-9857166904e8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:39.791509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 215/426] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:39 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:39.792879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4e1999-c3bb-4ba2-8f11-9376724b80aa None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:39.793157 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4e1999-c3bb-4ba2-8f11-9376724b80aa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:39.793528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4e1999-c3bb-4ba2-8f11-9376724b80aa None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:39.794985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 216/427] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:39.798119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4e1999-c3bb-4ba2-8f11-9376724b80aa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:39.862877 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7adcc4-d368-45e5-9ce2-63952a5e8248 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:39.893524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:39.909148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df426856-3462-42be-8539-8b228b9411ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:39.910093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df426856-3462-42be-8539-8b228b9411ae None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:39.910587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df426856-3462-42be-8539-8b228b9411ae None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:39.911408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df426856-3462-42be-8539-8b228b9411ae None admin] PATH_INFO: `/v3/projects/85be2a1b4ed54b91bfefe25b0945913e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:39.913033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df426856-3462-42be-8539-8b228b9411ae None admin] RBAC: Authorizing `identity:get_project(project_id=85be2a1b4ed54b91bfefe25b0945913e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:39.913242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:39.915644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df426856-3462-42be-8539-8b228b9411ae None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:39.921983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 212/428] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:39 2022] GET /identity/v3/projects/85be2a1b4ed54b91bfefe25b0945913e => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:39.923847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8dc34f-aaae-4d54-9e45-4f6396b6ae42 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:39.924414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8dc34f-aaae-4d54-9e45-4f6396b6ae42 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:39.924790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8dc34f-aaae-4d54-9e45-4f6396b6ae42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:39.927123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4e1999-c3bb-4ba2-8f11-9376724b80aa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:39.943688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ca8dc34f-aaae-4d54-9e45-4f6396b6ae42 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:39.945619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ca8dc34f-aaae-4d54-9e45-4f6396b6ae42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:39.968744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176bcfbf-543f-4c1b-a7ac-d88807a5492c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:39.969870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176bcfbf-543f-4c1b-a7ac-d88807a5492c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:39.970304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176bcfbf-543f-4c1b-a7ac-d88807a5492c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:39.970723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176bcfbf-543f-4c1b-a7ac-d88807a5492c None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:39.972413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176bcfbf-543f-4c1b-a7ac-d88807a5492c None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:39.974448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176bcfbf-543f-4c1b-a7ac-d88807a5492c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:39.982286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 217/429] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:39 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:39.982286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176bcfbf-543f-4c1b-a7ac-d88807a5492c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:40.026994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 213/430] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:40.028340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8dc34f-aaae-4d54-9e45-4f6396b6ae42 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:40.138982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176bcfbf-543f-4c1b-a7ac-d88807a5492c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:40.194852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8dc34f-aaae-4d54-9e45-4f6396b6ae42 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:40.225615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:40.228654 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b162778-5c20-4224-b07c-8e47ac71c21b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:40.229360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b162778-5c20-4224-b07c-8e47ac71c21b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:40.229635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b162778-5c20-4224-b07c-8e47ac71c21b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:40.229902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b162778-5c20-4224-b07c-8e47ac71c21b None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:40.231060 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b162778-5c20-4224-b07c-8e47ac71c21b None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:40.232489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b162778-5c20-4224-b07c-8e47ac71c21b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:40.234182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 218/431] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:39 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:40.235358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec8e66f-1dd3-4612-af76-7aa51a5d13e3 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:40.235639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec8e66f-1dd3-4612-af76-7aa51a5d13e3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:40.235915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec8e66f-1dd3-4612-af76-7aa51a5d13e3 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:40.237352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 219/432] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:40.237542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a8f41a-bcdb-4dc9-9e79-fbb8cf056d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:40.241883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a8f41a-bcdb-4dc9-9e79-fbb8cf056d39 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:40.241883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a8f41a-bcdb-4dc9-9e79-fbb8cf056d39 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:40.241883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a8f41a-bcdb-4dc9-9e79-fbb8cf056d39 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:40.241883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a8f41a-bcdb-4dc9-9e79-fbb8cf056d39 None admin] RBAC: Adding query filter params (group.id=0e21a7774468486e9c4b62328a0325f2, role.id=e4fa1b2d8c404336a855b6142fbf25d4, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:40.241883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a8f41a-bcdb-4dc9-9e79-fbb8cf056d39 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:40.241883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a8f41a-bcdb-4dc9-9e79-fbb8cf056d39 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:40.243784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec8e66f-1dd3-4612-af76-7aa51a5d13e3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:40.248635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 214/433] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 2 00:41:40 2022] GET /identity/v3/role_assignments?group.id=0e21a7774468486e9c4b62328a0325f2&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 611 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:40.249814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa96a97-8b12-4708-a1a3-9c36a528b367 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:40.250192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa96a97-8b12-4708-a1a3-9c36a528b367 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:40.250463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa96a97-8b12-4708-a1a3-9c36a528b367 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:40.275434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-daa96a97-8b12-4708-a1a3-9c36a528b367 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:40.276640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-daa96a97-8b12-4708-a1a3-9c36a528b367 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:40.307667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:40.374106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 215/434] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:40.374106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa96a97-8b12-4708-a1a3-9c36a528b367 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:40.435878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec8e66f-1dd3-4612-af76-7aa51a5d13e3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:40.483814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f06e9ca-4d85-491d-b5b1-ee29012a740e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:40.484419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f06e9ca-4d85-491d-b5b1-ee29012a740e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:40.484928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f06e9ca-4d85-491d-b5b1-ee29012a740e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:40.485199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f06e9ca-4d85-491d-b5b1-ee29012a740e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:40.486585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f06e9ca-4d85-491d-b5b1-ee29012a740e None admin] RBAC: Adding query filter params (group.id=0e21a7774468486e9c4b62328a0325f2, role.id=5e19a16e5bfd4c94b1c34ae717ccfc42, scope.project.id=85be2a1b4ed54b91bfefe25b0945913e) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:40.486896 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f06e9ca-4d85-491d-b5b1-ee29012a740e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:40.488324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f06e9ca-4d85-491d-b5b1-ee29012a740e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:40.497595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 220/435] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 2 00:41:40 2022] GET /identity/v3/role_assignments?group.id=0e21a7774468486e9c4b62328a0325f2&scope.project.id=85be2a1b4ed54b91bfefe25b0945913e&role.id=5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 611 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:40.532526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa96a97-8b12-4708-a1a3-9c36a528b367 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:40.558252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:40.580436 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb572345-41e8-455d-a957-ac086fd1b751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:40.581084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb572345-41e8-455d-a957-ac086fd1b751 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:40.581361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb572345-41e8-455d-a957-ac086fd1b751 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:40.581630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb572345-41e8-455d-a957-ac086fd1b751 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:40.594013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb572345-41e8-455d-a957-ac086fd1b751 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-bb572345-41e8-455d-a957-ac086fd1b751 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:40.600961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:40.601911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:40.603971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 216/436] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:40.608704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f06e9ca-4d85-491d-b5b1-ee29012a740e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:40.675087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb572345-41e8-455d-a957-ac086fd1b751 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:40.737700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:40.749662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f06e9ca-4d85-491d-b5b1-ee29012a740e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:40.786369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d40ba2-f2ed-4690-aa1b-680ccd42da05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:40.787757 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d40ba2-f2ed-4690-aa1b-680ccd42da05 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:40.788301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d40ba2-f2ed-4690-aa1b-680ccd42da05 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:40.788785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d40ba2-f2ed-4690-aa1b-680ccd42da05 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:40.789639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d40ba2-f2ed-4690-aa1b-680ccd42da05 None admin] RBAC: Adding query filter params (name=member) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:40.790673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d40ba2-f2ed-4690-aa1b-680ccd42da05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:40.792228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d40ba2-f2ed-4690-aa1b-680ccd42da05 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:40.800852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb572345-41e8-455d-a957-ac086fd1b751 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:40.805471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 221/437] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:41:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:40.811825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d40ba2-f2ed-4690-aa1b-680ccd42da05 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:40.848644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb854bbc-0f4b-461f-b0a4-831fa443ee1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:40.849418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb854bbc-0f4b-461f-b0a4-831fa443ee1f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:40.849574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb854bbc-0f4b-461f-b0a4-831fa443ee1f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:40.849818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb854bbc-0f4b-461f-b0a4-831fa443ee1f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:40.862097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb854bbc-0f4b-461f-b0a4-831fa443ee1f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-eb854bbc-0f4b-461f-b0a4-831fa443ee1f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:40.867351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:40.868832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:40.869640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:40.870337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 217/438] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:40.876267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb854bbc-0f4b-461f-b0a4-831fa443ee1f None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:40.929189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:40.945693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d40ba2-f2ed-4690-aa1b-680ccd42da05 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:40.987267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9121f755-aa11-4a67-a212-759f2303403e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:40.988124 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9121f755-aa11-4a67-a212-759f2303403e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:40.988336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9121f755-aa11-4a67-a212-759f2303403e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:40.988607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9121f755-aa11-4a67-a212-759f2303403e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:40.995761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb854bbc-0f4b-461f-b0a4-831fa443ee1f None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:41.015123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:41.049870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d972d323-505f-4a55-9610-dc0a7d1d0c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:41.050784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d972d323-505f-4a55-9610-dc0a7d1d0c85 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.051067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d972d323-505f-4a55-9610-dc0a7d1d0c85 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.051343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d972d323-505f-4a55-9610-dc0a7d1d0c85 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.052021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d972d323-505f-4a55-9610-dc0a7d1d0c85 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:41.052327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d972d323-505f-4a55-9610-dc0a7d1d0c85 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:41.053695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d972d323-505f-4a55-9610-dc0a7d1d0c85 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:41.067195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9121f755-aa11-4a67-a212-759f2303403e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:41.072411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 218/439] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:41.072607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9121f755-aa11-4a67-a212-759f2303403e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:41.075247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d972d323-505f-4a55-9610-dc0a7d1d0c85 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:41.201451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 222/440] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 02 00:41:41.202358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d972d323-505f-4a55-9610-dc0a7d1d0c85 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:41.202904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58570b56-4923-4a77-b099-c79c6fade908 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.203186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58570b56-4923-4a77-b099-c79c6fade908 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.203455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58570b56-4923-4a77-b099-c79c6fade908 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.204152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:41.209457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 223/441] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:41.209457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58570b56-4923-4a77-b099-c79c6fade908 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:41.246592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b027b90-8372-41b5-923e-4aa8da503066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:41.247196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b027b90-8372-41b5-923e-4aa8da503066 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.247467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b027b90-8372-41b5-923e-4aa8da503066 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.247734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b027b90-8372-41b5-923e-4aa8da503066 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.266702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:41.310969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58570b56-4923-4a77-b099-c79c6fade908 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:41.313296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b027b90-8372-41b5-923e-4aa8da503066 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:41.314845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b027b90-8372-41b5-923e-4aa8da503066 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:41.346277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bcc31a-f7bf-4668-b00f-e623fd15e2f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:41.346891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bcc31a-f7bf-4668-b00f-e623fd15e2f7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.347160 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bcc31a-f7bf-4668-b00f-e623fd15e2f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.347424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bcc31a-f7bf-4668-b00f-e623fd15e2f7 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.348514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bcc31a-f7bf-4668-b00f-e623fd15e2f7 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:41.349927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bcc31a-f7bf-4668-b00f-e623fd15e2f7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:41.351562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 224/442] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:41 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:41.353441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaca8e35-b6dc-4b1f-b370-b90ba117c02e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.353714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaca8e35-b6dc-4b1f-b370-b90ba117c02e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.353983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaca8e35-b6dc-4b1f-b370-b90ba117c02e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.371125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-aaca8e35-b6dc-4b1f-b370-b90ba117c02e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:41.372506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-aaca8e35-b6dc-4b1f-b370-b90ba117c02e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:41.414158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 219/443] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:41.415533 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b027b90-8372-41b5-923e-4aa8da503066 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:41.417765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:41.440468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 225/444] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:41.441363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaca8e35-b6dc-4b1f-b370-b90ba117c02e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:41.499480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:41.516257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b027b90-8372-41b5-923e-4aa8da503066 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:41.547413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaca8e35-b6dc-4b1f-b370-b90ba117c02e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:41.551411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7219f3-e6ab-4f78-81d9-e420d9c797c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:41.552032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7219f3-e6ab-4f78-81d9-e420d9c797c5 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.552307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7219f3-e6ab-4f78-81d9-e420d9c797c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.552577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7219f3-e6ab-4f78-81d9-e420d9c797c5 None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.553686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7219f3-e6ab-4f78-81d9-e420d9c797c5 None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:41.555152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d7219f3-e6ab-4f78-81d9-e420d9c797c5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:41.556772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 220/445] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:41 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:41.562343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7219f3-e6ab-4f78-81d9-e420d9c797c5 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:41.590281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafc6470-cdca-4514-aaa2-a65ea642a548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:41.591251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafc6470-cdca-4514-aaa2-a65ea642a548 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.591789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafc6470-cdca-4514-aaa2-a65ea642a548 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.592064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafc6470-cdca-4514-aaa2-a65ea642a548 None admin] PATH_INFO: `/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.593242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafc6470-cdca-4514-aaa2-a65ea642a548 None admin] RBAC: Authorizing `identity:get_project(project_id=74b104206ad84d6bbb9d5fd926b6f1aa)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:41.595248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafc6470-cdca-4514-aaa2-a65ea642a548 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:41.602098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 226/446] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:41 2022] GET /identity/v3/projects/74b104206ad84d6bbb9d5fd926b6f1aa => generated 302 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:41.603131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafc6470-cdca-4514-aaa2-a65ea642a548 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:41.630729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:41.674864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d7219f3-e6ab-4f78-81d9-e420d9c797c5 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:41.715967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ada3810-1cb7-4953-b6d2-635a064ea3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:41.716584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ada3810-1cb7-4953-b6d2-635a064ea3e1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.716670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafc6470-cdca-4514-aaa2-a65ea642a548 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:41.716831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ada3810-1cb7-4953-b6d2-635a064ea3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.717098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ada3810-1cb7-4953-b6d2-635a064ea3e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.717816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ada3810-1cb7-4953-b6d2-635a064ea3e1 None admin] RBAC: Adding query filter params (group.id=0e21a7774468486e9c4b62328a0325f2, role.id=e4fa1b2d8c404336a855b6142fbf25d4, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:41.718151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ada3810-1cb7-4953-b6d2-635a064ea3e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:41.719593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ada3810-1cb7-4953-b6d2-635a064ea3e1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:41.725603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 221/447] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 2 00:41:41 2022] GET /identity/v3/role_assignments?group.id=0e21a7774468486e9c4b62328a0325f2&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=e4fa1b2d8c404336a855b6142fbf25d4 => generated 611 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:41.725603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ada3810-1cb7-4953-b6d2-635a064ea3e1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:41.742932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:41.756982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d651011-c494-489f-bb4e-e7cda60480ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:41.757539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d651011-c494-489f-bb4e-e7cda60480ac None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.757814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d651011-c494-489f-bb4e-e7cda60480ac None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.758111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d651011-c494-489f-bb4e-e7cda60480ac None admin] PATH_INFO: `/v3/groups/0e21a7774468486e9c4b62328a0325f2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.759595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d651011-c494-489f-bb4e-e7cda60480ac None admin] RBAC: Authorizing `identity:get_group(group_id=0e21a7774468486e9c4b62328a0325f2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:41.761071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d651011-c494-489f-bb4e-e7cda60480ac None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:41.767554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 227/448] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:41 2022] GET /identity/v3/groups/0e21a7774468486e9c4b62328a0325f2 => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:41.768876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d651011-c494-489f-bb4e-e7cda60480ac None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:41.852522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ada3810-1cb7-4953-b6d2-635a064ea3e1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:41.883671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d651011-c494-489f-bb4e-e7cda60480ac None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:41.894229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38937e5b-586b-49c0-9b8c-dc9903020f54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:41.895219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38937e5b-586b-49c0-9b8c-dc9903020f54 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.895514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38937e5b-586b-49c0-9b8c-dc9903020f54 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.896066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38937e5b-586b-49c0-9b8c-dc9903020f54 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.898011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:41.909194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38937e5b-586b-49c0-9b8c-dc9903020f54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-38937e5b-586b-49c0-9b8c-dc9903020f54 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:41.914463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:41.915432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:41.916254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 222/449] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:41:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:41.921515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38937e5b-586b-49c0-9b8c-dc9903020f54 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:41.926950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c397f678-df2e-41fe-ac48-c05f5e7e9602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:41.927790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c397f678-df2e-41fe-ac48-c05f5e7e9602 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:41.928249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c397f678-df2e-41fe-ac48-c05f5e7e9602 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:41.928686 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c397f678-df2e-41fe-ac48-c05f5e7e9602 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:41.929573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c397f678-df2e-41fe-ac48-c05f5e7e9602 None admin] RBAC: Adding query filter params (group.id=0e21a7774468486e9c4b62328a0325f2, role.id=5e19a16e5bfd4c94b1c34ae717ccfc42, scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:41.930834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c397f678-df2e-41fe-ac48-c05f5e7e9602 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:41.932439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c397f678-df2e-41fe-ac48-c05f5e7e9602 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:41.939268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 228/450] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 2 00:41:41 2022] GET /identity/v3/role_assignments?group.id=0e21a7774468486e9c4b62328a0325f2&scope.project.id=74b104206ad84d6bbb9d5fd926b6f1aa&role.id=5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 611 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:42.041067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38937e5b-586b-49c0-9b8c-dc9903020f54 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:42.086253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b09319d-f8ff-4be7-b41e-0d31eadb77da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:42.086925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09319d-f8ff-4be7-b41e-0d31eadb77da None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:42.087199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09319d-f8ff-4be7-b41e-0d31eadb77da None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:42.087468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b09319d-f8ff-4be7-b41e-0d31eadb77da None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:42.088145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b09319d-f8ff-4be7-b41e-0d31eadb77da None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:42.088445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b09319d-f8ff-4be7-b41e-0d31eadb77da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:42.089853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b09319d-f8ff-4be7-b41e-0d31eadb77da None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:42.099484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 223/451] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:42.105789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c397f678-df2e-41fe-ac48-c05f5e7e9602 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:42.169419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:42.217703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c397f678-df2e-41fe-ac48-c05f5e7e9602 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:42.249770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dadbb2-33ef-4a82-9c15-a71b129ab023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:42.250402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dadbb2-33ef-4a82-9c15-a71b129ab023 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:42.250679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dadbb2-33ef-4a82-9c15-a71b129ab023 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:42.250942 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dadbb2-33ef-4a82-9c15-a71b129ab023 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:42.321584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dadbb2-33ef-4a82-9c15-a71b129ab023 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:42.323053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dadbb2-33ef-4a82-9c15-a71b129ab023 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:42.378175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:42.421685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 229/452] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 02 00:41:42.427339 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b09319d-f8ff-4be7-b41e-0d31eadb77da None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:42.480700 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:42.522914 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b09319d-f8ff-4be7-b41e-0d31eadb77da None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:42.553608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8246c1c-8bfa-4865-a70b-0ce398db713c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:42.554246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8246c1c-8bfa-4865-a70b-0ce398db713c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:42.554522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8246c1c-8bfa-4865-a70b-0ce398db713c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:42.554792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8246c1c-8bfa-4865-a70b-0ce398db713c None admin] PATH_INFO: `/v3/projects/2ea30d4949414e4cafefc43b9507a1d0` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:42.556379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8246c1c-8bfa-4865-a70b-0ce398db713c None admin] RBAC: Authorizing `identity:get_project(project_id=2ea30d4949414e4cafefc43b9507a1d0)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:42.557814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8246c1c-8bfa-4865-a70b-0ce398db713c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:42.559386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 224/453] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:41:42 2022] GET /identity/v3/projects/2ea30d4949414e4cafefc43b9507a1d0 => generated 344 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:42.564687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dadbb2-33ef-4a82-9c15-a71b129ab023 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:42.660951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dadbb2-33ef-4a82-9c15-a71b129ab023 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:42.694182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eefe3c9-c3a6-4eb0-b2e0-7fc80ab04f20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:42.694789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eefe3c9-c3a6-4eb0-b2e0-7fc80ab04f20 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:42.695058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eefe3c9-c3a6-4eb0-b2e0-7fc80ab04f20 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:42.695320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eefe3c9-c3a6-4eb0-b2e0-7fc80ab04f20 None admin] PATH_INFO: `/v3/groups/ee3ab5fff6fd48a587a82854986f7865` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:42.696999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eefe3c9-c3a6-4eb0-b2e0-7fc80ab04f20 None admin] RBAC: Authorizing `identity:get_group(group_id=ee3ab5fff6fd48a587a82854986f7865)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:42.698461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eefe3c9-c3a6-4eb0-b2e0-7fc80ab04f20 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:42.699977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 230/454] 10.222.0.25 () {56 vars in 1180 bytes} [Sun Oct 2 00:41:42 2022] GET /identity/v3/groups/ee3ab5fff6fd48a587a82854986f7865 => generated 229 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:42.705848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8246c1c-8bfa-4865-a70b-0ce398db713c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:42.800563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8246c1c-8bfa-4865-a70b-0ce398db713c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:42.822569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:42.832738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfe06b8-785c-44ca-8ab1-c9df684b801c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:42.833320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfe06b8-785c-44ca-8ab1-c9df684b801c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:42.833599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfe06b8-785c-44ca-8ab1-c9df684b801c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:42.833869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfe06b8-785c-44ca-8ab1-c9df684b801c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:42.834623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfe06b8-785c-44ca-8ab1-c9df684b801c None admin] RBAC: Adding query filter params (group.id=ee3ab5fff6fd48a587a82854986f7865, role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=2ea30d4949414e4cafefc43b9507a1d0) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:42.834930 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfe06b8-785c-44ca-8ab1-c9df684b801c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:42.836335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfe06b8-785c-44ca-8ab1-c9df684b801c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:42.842425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 225/455] 10.222.0.25 () {56 vars in 1376 bytes} [Sun Oct 2 00:41:42 2022] GET /identity/v3/role_assignments?group.id=ee3ab5fff6fd48a587a82854986f7865&scope.project.id=2ea30d4949414e4cafefc43b9507a1d0&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 611 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:45.472443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d75036-007e-4aae-8cd0-8a4b919f0b7d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:45.472923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d75036-007e-4aae-8cd0-8a4b919f0b7d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:45.472923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d75036-007e-4aae-8cd0-8a4b919f0b7d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:45.474477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 231/456] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:45.483907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f874040d-effd-46cc-84e4-174216952310 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:45.484359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f874040d-effd-46cc-84e4-174216952310 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:45.484435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f874040d-effd-46cc-84e4-174216952310 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:45.511329 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f874040d-effd-46cc-84e4-174216952310 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:45.512575 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f874040d-effd-46cc-84e4-174216952310 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:45.608538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 226/457] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:45.673674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18076917-76f3-457c-aa73-37a0d59caf91 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:45.674195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18076917-76f3-457c-aa73-37a0d59caf91 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:45.674628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18076917-76f3-457c-aa73-37a0d59caf91 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:45.676324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 232/458] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:45.682136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbb322b-c5f6-402c-8bbd-5a3715fa0e29 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:45.682424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbb322b-c5f6-402c-8bbd-5a3715fa0e29 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:45.682696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbb322b-c5f6-402c-8bbd-5a3715fa0e29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:45.711026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-adbb322b-c5f6-402c-8bbd-5a3715fa0e29 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:45.712335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-adbb322b-c5f6-402c-8bbd-5a3715fa0e29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:45.804561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 227/459] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:45.930306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18076917-76f3-457c-aa73-37a0d59caf91 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:45.960330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:46.003962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9e4244-239c-41cf-a584-0d9063f22b85 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.004512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9e4244-239c-41cf-a584-0d9063f22b85 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.004931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9e4244-239c-41cf-a584-0d9063f22b85 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.006796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 228/460] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:46.012826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b57316c-9a53-40ce-97f6-e22c19fadb26 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.013361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b57316c-9a53-40ce-97f6-e22c19fadb26 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.013849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b57316c-9a53-40ce-97f6-e22c19fadb26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.061180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9b57316c-9a53-40ce-97f6-e22c19fadb26 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:46.062317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9b57316c-9a53-40ce-97f6-e22c19fadb26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:46.083735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18076917-76f3-457c-aa73-37a0d59caf91 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:46.135821 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a5b585-63d0-4d0f-91e5-da75d18f5c86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:46.137261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a5b585-63d0-4d0f-91e5-da75d18f5c86 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.137539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a5b585-63d0-4d0f-91e5-da75d18f5c86 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.137813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a5b585-63d0-4d0f-91e5-da75d18f5c86 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.146609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:46.150085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a5b585-63d0-4d0f-91e5-da75d18f5c86 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:46.152149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a5b585-63d0-4d0f-91e5-da75d18f5c86 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-88a5b585-63d0-4d0f-91e5-da75d18f5c86 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:46.171886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:46.172813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:46.173667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:46.173667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:46.173667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:46.174294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 229/461] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:46.174294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b57316c-9a53-40ce-97f6-e22c19fadb26 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:46.178201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 233/462] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:46.188217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a5b585-63d0-4d0f-91e5-da75d18f5c86 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:46.332112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b57316c-9a53-40ce-97f6-e22c19fadb26 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:46.362085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:46.372688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc23e2be-3ec5-4f51-9b9f-f1f6237ba786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:46.373433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc23e2be-3ec5-4f51-9b9f-f1f6237ba786 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.373715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc23e2be-3ec5-4f51-9b9f-f1f6237ba786 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.373989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc23e2be-3ec5-4f51-9b9f-f1f6237ba786 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.385351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc23e2be-3ec5-4f51-9b9f-f1f6237ba786 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:46.387052 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc23e2be-3ec5-4f51-9b9f-f1f6237ba786 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:46.389585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a5b585-63d0-4d0f-91e5-da75d18f5c86 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-fc23e2be-3ec5-4f51-9b9f-f1f6237ba786 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:46.403530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:46.404475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:46.405321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:46.405321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:46.405321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:46.405321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 230/463] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:46.406472 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eed88f-3f10-4005-9b4f-3a8066068527 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.406573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eed88f-3f10-4005-9b4f-3a8066068527 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.406851 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59eed88f-3f10-4005-9b4f-3a8066068527 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.408069 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 231/464] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:46.412840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eed88f-3f10-4005-9b4f-3a8066068527 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:46.428269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbef42ba-71a6-4853-8f35-372d5f401926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:46.429287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbef42ba-71a6-4853-8f35-372d5f401926 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.430653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbef42ba-71a6-4853-8f35-372d5f401926 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.431412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbef42ba-71a6-4853-8f35-372d5f401926 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.432258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbef42ba-71a6-4853-8f35-372d5f401926 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:46.432884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbef42ba-71a6-4853-8f35-372d5f401926 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:46.434805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbef42ba-71a6-4853-8f35-372d5f401926 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:46.446073 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:46.462389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 234/465] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:46.464609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba24c8da-bfd9-4c2f-8646-6bd7a387c2fc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.465111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba24c8da-bfd9-4c2f-8646-6bd7a387c2fc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.465585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba24c8da-bfd9-4c2f-8646-6bd7a387c2fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.483900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ba24c8da-bfd9-4c2f-8646-6bd7a387c2fc None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:46.485101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ba24c8da-bfd9-4c2f-8646-6bd7a387c2fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:46.525380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59eed88f-3f10-4005-9b4f-3a8066068527 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:46.555773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 235/466] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:46.556975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba24c8da-bfd9-4c2f-8646-6bd7a387c2fc None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:46.572919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a2928c-5992-4b06-b9c7-c0f57971022f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:46.573831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2928c-5992-4b06-b9c7-c0f57971022f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.574351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2928c-5992-4b06-b9c7-c0f57971022f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.574882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a2928c-5992-4b06-b9c7-c0f57971022f None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.575846 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a2928c-5992-4b06-b9c7-c0f57971022f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:46.576336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a2928c-5992-4b06-b9c7-c0f57971022f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:46.578036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a2928c-5992-4b06-b9c7-c0f57971022f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:46.591720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 232/467] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:46.592893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a2928c-5992-4b06-b9c7-c0f57971022f None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:46.653380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:46.697743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:46.707015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba24c8da-bfd9-4c2f-8646-6bd7a387c2fc None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:46.709189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a2928c-5992-4b06-b9c7-c0f57971022f None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:46.747751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce471a78-f38e-4caf-b110-1c2f69a4f43a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:46.748603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce471a78-f38e-4caf-b110-1c2f69a4f43a None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.748960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce471a78-f38e-4caf-b110-1c2f69a4f43a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.749306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce471a78-f38e-4caf-b110-1c2f69a4f43a None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.750242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce471a78-f38e-4caf-b110-1c2f69a4f43a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:46.752041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce471a78-f38e-4caf-b110-1c2f69a4f43a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:46.752394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2315c1-4913-4d1e-95a0-de604ffd59a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:46.753200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2315c1-4913-4d1e-95a0-de604ffd59a2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.753641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2315c1-4913-4d1e-95a0-de604ffd59a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.754094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2315c1-4913-4d1e-95a0-de604ffd59a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.767649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2315c1-4913-4d1e-95a0-de604ffd59a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:46.769751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2315c1-4913-4d1e-95a0-de604ffd59a2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:46.779519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 233/468] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:41:46 2022] POST /identity/v3/users => generated 246 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:41:46.780620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce471a78-f38e-4caf-b110-1c2f69a4f43a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fb2315c1-4913-4d1e-95a0-de604ffd59a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:46.789230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:46.790266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:46.791093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:46.791093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:46.791093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:46.793345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 236/469] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:46.801878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2315c1-4913-4d1e-95a0-de604ffd59a2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:46.905006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce471a78-f38e-4caf-b110-1c2f69a4f43a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:46.948404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae5547-ae28-4219-af33-b340e26262c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:46.949359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae5547-ae28-4219-af33-b340e26262c6 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:46.949839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae5547-ae28-4219-af33-b340e26262c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:46.950144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2315c1-4913-4d1e-95a0-de604ffd59a2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:46.950451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae5547-ae28-4219-af33-b340e26262c6 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:46.951250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ae5547-ae28-4219-af33-b340e26262c6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:46.954250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ae5547-ae28-4219-af33-b340e26262c6 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:46.969825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:46.990083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 234/470] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:41:46 2022] POST /identity/v3/users => generated 248 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:41:46.990083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae5547-ae28-4219-af33-b340e26262c6 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:46.990083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:47.007592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1644af9-3575-4f8d-bfd8-a5c9035b4088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:47.008393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1644af9-3575-4f8d-bfd8-a5c9035b4088 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:47.008998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1644af9-3575-4f8d-bfd8-a5c9035b4088 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:47.009341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1644af9-3575-4f8d-bfd8-a5c9035b4088 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:47.010237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1644af9-3575-4f8d-bfd8-a5c9035b4088 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:47.010647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1644af9-3575-4f8d-bfd8-a5c9035b4088 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:47.012279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1644af9-3575-4f8d-bfd8-a5c9035b4088 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:47.038114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 237/471] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:47.038114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c04a7a-e517-42ba-a7fd-89a311dc7def None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:47.038114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c04a7a-e517-42ba-a7fd-89a311dc7def None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:47.038114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c04a7a-e517-42ba-a7fd-89a311dc7def None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:47.038114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 238/472] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:47.042574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c04a7a-e517-42ba-a7fd-89a311dc7def None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:47.145913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae5547-ae28-4219-af33-b340e26262c6 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:47.178394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c04a7a-e517-42ba-a7fd-89a311dc7def None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:47.194467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a169c02a-02df-41b9-b2b2-acc72de2c369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:47.195568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a169c02a-02df-41b9-b2b2-acc72de2c369 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:47.196496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a169c02a-02df-41b9-b2b2-acc72de2c369 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:47.196864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a169c02a-02df-41b9-b2b2-acc72de2c369 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:47.210806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a169c02a-02df-41b9-b2b2-acc72de2c369 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:47.212408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a169c02a-02df-41b9-b2b2-acc72de2c369 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:47.223362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57129163-3aee-4ae8-9dc4-18dc8444846c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:47.223970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57129163-3aee-4ae8-9dc4-18dc8444846c None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:47.224255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57129163-3aee-4ae8-9dc4-18dc8444846c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:47.224508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57129163-3aee-4ae8-9dc4-18dc8444846c None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:47.225144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57129163-3aee-4ae8-9dc4-18dc8444846c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:47.227453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57129163-3aee-4ae8-9dc4-18dc8444846c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-a169c02a-02df-41b9-b2b2-acc72de2c369 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:47.228895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:47.229725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:47.230664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:47.230664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:47.230664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:47.230664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 235/473] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:47.232311 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a827371-4686-41ec-acb7-e3d5a97f1c7b None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:47.232311 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a827371-4686-41ec-acb7-e3d5a97f1c7b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:47.232311 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a827371-4686-41ec-acb7-e3d5a97f1c7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:47.255991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 239/474] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:41:47 2022] POST /identity/v3/users => generated 248 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:41:47.256372 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57129163-3aee-4ae8-9dc4-18dc8444846c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:47.258606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-7a827371-4686-41ec-acb7-e3d5a97f1c7b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:47.258996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7a827371-4686-41ec-acb7-e3d5a97f1c7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:47.303660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:47.332118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:47.355801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 236/475] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:47.393372 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57129163-3aee-4ae8-9dc4-18dc8444846c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:47.431678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34339cd6-a138-45b8-854f-7e3f7ef34857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:47.432240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34339cd6-a138-45b8-854f-7e3f7ef34857 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:47.432521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34339cd6-a138-45b8-854f-7e3f7ef34857 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:47.432788 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34339cd6-a138-45b8-854f-7e3f7ef34857 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:47.433437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34339cd6-a138-45b8-854f-7e3f7ef34857 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:47.433739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34339cd6-a138-45b8-854f-7e3f7ef34857 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:47.435232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34339cd6-a138-45b8-854f-7e3f7ef34857 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:47.449196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 240/476] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:47.455161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a827371-4686-41ec-acb7-e3d5a97f1c7b None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:47.557978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a827371-4686-41ec-acb7-e3d5a97f1c7b None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:47.592440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676bfa57-e686-42a1-97e2-d7bdf21013d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:47.593044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676bfa57-e686-42a1-97e2-d7bdf21013d1 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:47.593315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676bfa57-e686-42a1-97e2-d7bdf21013d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:47.593632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676bfa57-e686-42a1-97e2-d7bdf21013d1 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:47.594304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676bfa57-e686-42a1-97e2-d7bdf21013d1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:47.595764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676bfa57-e686-42a1-97e2-d7bdf21013d1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:47.620644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 237/477] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:41:47 2022] POST /identity/v3/users => generated 249 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:41:47.684152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34339cd6-a138-45b8-854f-7e3f7ef34857 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:47.792268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:47.801916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34339cd6-a138-45b8-854f-7e3f7ef34857 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:47.840043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eeec2f-36ed-4981-95ef-4b2fb4e7e861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:47.840997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eeec2f-36ed-4981-95ef-4b2fb4e7e861 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:47.841470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eeec2f-36ed-4981-95ef-4b2fb4e7e861 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:47.841932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eeec2f-36ed-4981-95ef-4b2fb4e7e861 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:47.842826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49eeec2f-36ed-4981-95ef-4b2fb4e7e861 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:47.844414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49eeec2f-36ed-4981-95ef-4b2fb4e7e861 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:47.852726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.sql.core [None req-49eeec2f-36ed-4981-95ef-4b2fb4e7e861 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 02 00:41:47.852726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 02 00:41:47.852726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [parameters: {'id': 'f9f5e7314ec84dc2b8f030be2938ca75', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 02 00:41:47.852726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81433) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-49eeec2f-36ed-4981-95ef-4b2fb4e7e861 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = self._query(query) Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application conn.query(q) Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result.read() Oct 02 00:41:47.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return ref.to_dict() Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 02 00:41:47.870658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application next(self.gen) Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application yield resource Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application next(self.gen) Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.session.rollback() Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.force_reraise() Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise self.value Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application session.commit() Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 02 00:41:47.872061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._prepare_impl() Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.session.flush() Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._flush(objects) Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application compat.raise_( Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application flush_context.execute() Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rec.execute(self) Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 02 00:41:47.873121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 02 00:41:47.874424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = self._query(query) Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application conn.query(q) Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result.read() Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 02 00:41:47.876551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [parameters: {'id': 'f9f5e7314ec84dc2b8f030be2938ca75', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:47.882955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 02 00:41:47.886290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:47.888324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 241/478] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Oct 2 00:41:47 2022] POST /identity/v3/roles => generated 144 bytes in 188 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:47.888462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676bfa57-e686-42a1-97e2-d7bdf21013d1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:47.930640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:47.996125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676bfa57-e686-42a1-97e2-d7bdf21013d1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:48.033183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf12200-7c6e-45e4-971d-d4f9df23c670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:48.034034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf12200-7c6e-45e4-971d-d4f9df23c670 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:48.034134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf12200-7c6e-45e4-971d-d4f9df23c670 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:48.034424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf12200-7c6e-45e4-971d-d4f9df23c670 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:48.045665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf12200-7c6e-45e4-971d-d4f9df23c670 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-4bf12200-7c6e-45e4-971d-d4f9df23c670 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:48.051002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:48.051898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:48.053100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 238/479] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:41:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:48.058376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eeec2f-36ed-4981-95ef-4b2fb4e7e861 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:48.171863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eeec2f-36ed-4981-95ef-4b2fb4e7e861 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:48.212664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1af627-c9e4-460c-9190-5144014657bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:48.213647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1af627-c9e4-460c-9190-5144014657bf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:48.214171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1af627-c9e4-460c-9190-5144014657bf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:48.214635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1af627-c9e4-460c-9190-5144014657bf None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:48.215318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1af627-c9e4-460c-9190-5144014657bf None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:48.215626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1af627-c9e4-460c-9190-5144014657bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:48.217002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1af627-c9e4-460c-9190-5144014657bf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:48.226239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 242/480] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:49.299049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84287eca-d2b8-4f18-8937-a4850e7671c8 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:49.299526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84287eca-d2b8-4f18-8937-a4850e7671c8 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:49.299526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84287eca-d2b8-4f18-8937-a4850e7671c8 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:49.300982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 239/481] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:49.309760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08970b64-26b6-4a03-8655-5ee1c10f6674 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:49.310173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08970b64-26b6-4a03-8655-5ee1c10f6674 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:49.310397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08970b64-26b6-4a03-8655-5ee1c10f6674 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:49.332543 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-08970b64-26b6-4a03-8655-5ee1c10f6674 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:49.333777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-08970b64-26b6-4a03-8655-5ee1c10f6674 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:49.350568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:49.424649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 243/482] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:49.616079 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae5b9e7-c0b1-44c1-a2f0-3fb996f3ea2d None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:49.616409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae5b9e7-c0b1-44c1-a2f0-3fb996f3ea2d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:49.616610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae5b9e7-c0b1-44c1-a2f0-3fb996f3ea2d None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:49.618139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 240/483] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:49.623506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a04d20-b37a-438f-8a0c-2e43eff5c864 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:49.623794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a04d20-b37a-438f-8a0c-2e43eff5c864 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:49.624067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a04d20-b37a-438f-8a0c-2e43eff5c864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:49.647533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-31a04d20-b37a-438f-8a0c-2e43eff5c864 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:49.648793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-31a04d20-b37a-438f-8a0c-2e43eff5c864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:49.739877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 244/484] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:49.811189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae5b9e7-c0b1-44c1-a2f0-3fb996f3ea2d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:49.935216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:49.939639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae5b9e7-c0b1-44c1-a2f0-3fb996f3ea2d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:49.987292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a610c0cc-dec1-4e30-805f-8425681408e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:49.987923 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a610c0cc-dec1-4e30-805f-8425681408e8 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:49.988197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a610c0cc-dec1-4e30-805f-8425681408e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:49.988463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a610c0cc-dec1-4e30-805f-8425681408e8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.000370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a610c0cc-dec1-4e30-805f-8425681408e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-a610c0cc-dec1-4e30-805f-8425681408e8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:50.005039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:50.005908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:50.006760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 241/485] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:41:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:50.015149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a04d20-b37a-438f-8a0c-2e43eff5c864 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:50.098339 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a610c0cc-dec1-4e30-805f-8425681408e8 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:50.135365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:50.156963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a04d20-b37a-438f-8a0c-2e43eff5c864 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:50.193417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a9bcd3-f883-450e-99aa-d7e3c7ae97d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:50.194270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a9bcd3-f883-450e-99aa-d7e3c7ae97d3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:50.194709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a9bcd3-f883-450e-99aa-d7e3c7ae97d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:50.195239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a9bcd3-f883-450e-99aa-d7e3c7ae97d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.196167 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a9bcd3-f883-450e-99aa-d7e3c7ae97d3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:50.196671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a9bcd3-f883-450e-99aa-d7e3c7ae97d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:50.198260 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a9bcd3-f883-450e-99aa-d7e3c7ae97d3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:50.207083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 245/486] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:41:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:50.208598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2d7495-17b7-4683-a445-6e5c9d516127 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:50.209072 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2d7495-17b7-4683-a445-6e5c9d516127 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:50.209548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2d7495-17b7-4683-a445-6e5c9d516127 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.210992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 246/487] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:50.212924 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2d7495-17b7-4683-a445-6e5c9d516127 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:50.269015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a610c0cc-dec1-4e30-805f-8425681408e8 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:50.315732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc364963-a5ae-4dd1-825a-7ab02004c553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:50.316379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc364963-a5ae-4dd1-825a-7ab02004c553 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:50.316631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc364963-a5ae-4dd1-825a-7ab02004c553 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:50.316899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc364963-a5ae-4dd1-825a-7ab02004c553 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.322547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2d7495-17b7-4683-a445-6e5c9d516127 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:50.329459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc364963-a5ae-4dd1-825a-7ab02004c553 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-dc364963-a5ae-4dd1-825a-7ab02004c553 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:50.337827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:50.338804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:50.339608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 242/488] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:41:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:50.340758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5eba0a-0d10-45ec-b6ce-9caa9d6e8b77 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:50.340758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5eba0a-0d10-45ec-b6ce-9caa9d6e8b77 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:50.341467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5eba0a-0d10-45ec-b6ce-9caa9d6e8b77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.362895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb00157-8d54-4151-bf1a-e65ed8e289e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:50.363296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-3b5eba0a-0d10-45ec-b6ce-9caa9d6e8b77 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:50.364088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb00157-8d54-4151-bf1a-e65ed8e289e9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:50.364495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3b5eba0a-0d10-45ec-b6ce-9caa9d6e8b77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:50.364759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb00157-8d54-4151-bf1a-e65ed8e289e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:50.365212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb00157-8d54-4151-bf1a-e65ed8e289e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.374402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:50.376814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb00157-8d54-4151-bf1a-e65ed8e289e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:50.378600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb00157-8d54-4151-bf1a-e65ed8e289e9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:50.386099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-3cb00157-8d54-4151-bf1a-e65ed8e289e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:50.397722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:50.398777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:50.399621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:50.399621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:50.399621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:50.401402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 247/489] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:50.402488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb00157-8d54-4151-bf1a-e65ed8e289e9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:50.446228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 243/490] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:50.447038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5eba0a-0d10-45ec-b6ce-9caa9d6e8b77 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:50.556098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb00157-8d54-4151-bf1a-e65ed8e289e9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:50.584306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5eba0a-0d10-45ec-b6ce-9caa9d6e8b77 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:50.620491 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1a8ea0-3995-4078-993a-caedc7012cc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:50.622512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1a8ea0-3995-4078-993a-caedc7012cc2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:50.626545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1a8ea0-3995-4078-993a-caedc7012cc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:50.627040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1a8ea0-3995-4078-993a-caedc7012cc2 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.627903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1a8ea0-3995-4078-993a-caedc7012cc2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:50.628375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1a8ea0-3995-4078-993a-caedc7012cc2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:50.629951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1a8ea0-3995-4078-993a-caedc7012cc2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:50.655779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 248/491] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:41:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:50.659662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c3c85b-e1b4-4171-8e7b-1d217978f2ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:50.660293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c3c85b-e1b4-4171-8e7b-1d217978f2ed None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:50.660570 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c3c85b-e1b4-4171-8e7b-1d217978f2ed None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:50.660841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c3c85b-e1b4-4171-8e7b-1d217978f2ed None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.661488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c3c85b-e1b4-4171-8e7b-1d217978f2ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:50.662518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1a8ea0-3995-4078-993a-caedc7012cc2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:50.663564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c3c85b-e1b4-4171-8e7b-1d217978f2ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:50.665518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c3c85b-e1b4-4171-8e7b-1d217978f2ed None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:50.689174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 244/492] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:50.694894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c3c85b-e1b4-4171-8e7b-1d217978f2ed None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:50.730199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:50.753975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:50.805710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1a8ea0-3995-4078-993a-caedc7012cc2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:50.817353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c3c85b-e1b4-4171-8e7b-1d217978f2ed None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:50.842184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f347df-94d5-4065-8d8e-7b8b4a751f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:50.842919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f347df-94d5-4065-8d8e-7b8b4a751f2c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:50.843197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f347df-94d5-4065-8d8e-7b8b4a751f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:50.843584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f347df-94d5-4065-8d8e-7b8b4a751f2c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.851001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693cfc88-e49f-40d9-9a40-d3dc21bee8b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:50.851652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693cfc88-e49f-40d9-9a40-d3dc21bee8b6 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:50.851930 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693cfc88-e49f-40d9-9a40-d3dc21bee8b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:50.852204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693cfc88-e49f-40d9-9a40-d3dc21bee8b6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.855327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f347df-94d5-4065-8d8e-7b8b4a751f2c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:50.857066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f347df-94d5-4065-8d8e-7b8b4a751f2c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:50.859793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693cfc88-e49f-40d9-9a40-d3dc21bee8b6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:50.861215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693cfc88-e49f-40d9-9a40-d3dc21bee8b6 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e2f347df-94d5-4065-8d8e-7b8b4a751f2c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:50.872794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:50.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:50.874630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:50.874630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:50.874630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-693cfc88-e49f-40d9-9a40-d3dc21bee8b6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:41:50.874805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:50.875626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:50.876546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 245/493] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 2 00:41:50 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:50.877107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cb2e50-00e6-4a7b-b913-e3ee8c42f53c None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:50.877107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cb2e50-00e6-4a7b-b913-e3ee8c42f53c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:50.877229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cb2e50-00e6-4a7b-b913-e3ee8c42f53c None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:50.878385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 246/494] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:50 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:50.885663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 249/495] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:50.885663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f347df-94d5-4065-8d8e-7b8b4a751f2c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:50.886020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cb2e50-00e6-4a7b-b913-e3ee8c42f53c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:50.962700 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:50.987217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f347df-94d5-4065-8d8e-7b8b4a751f2c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:50.995004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:50.997736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cb2e50-00e6-4a7b-b913-e3ee8c42f53c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:51.025080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779e0ba5-c384-4e75-b7e5-8c1b69db1053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:51.025687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e0ba5-c384-4e75-b7e5-8c1b69db1053 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.025969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e0ba5-c384-4e75-b7e5-8c1b69db1053 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.026335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779e0ba5-c384-4e75-b7e5-8c1b69db1053 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.027054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779e0ba5-c384-4e75-b7e5-8c1b69db1053 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:51.027364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779e0ba5-c384-4e75-b7e5-8c1b69db1053 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:51.028819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779e0ba5-c384-4e75-b7e5-8c1b69db1053 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:51.040110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0928a267-8321-4d04-b8a6-b3617e4d16a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:51.040747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0928a267-8321-4d04-b8a6-b3617e4d16a3 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.041055 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0928a267-8321-4d04-b8a6-b3617e4d16a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.041337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0928a267-8321-4d04-b8a6-b3617e4d16a3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.052858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0928a267-8321-4d04-b8a6-b3617e4d16a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-0928a267-8321-4d04-b8a6-b3617e4d16a3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:51.058188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:51.059096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:51.059900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 247/496] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:41:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:51.060561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8bfc2e-e2d2-4d98-a72f-48b15576cc00 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.061359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8bfc2e-e2d2-4d98-a72f-48b15576cc00 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.061359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8bfc2e-e2d2-4d98-a72f-48b15576cc00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.071030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 250/497] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 2 00:41:50 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:51.072117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779e0ba5-c384-4e75-b7e5-8c1b69db1053 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:51.080247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-0f8bfc2e-e2d2-4d98-a72f-48b15576cc00 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:51.081436 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0f8bfc2e-e2d2-4d98-a72f-48b15576cc00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:51.157030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 248/498] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:51.158167 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8bfc2e-e2d2-4d98-a72f-48b15576cc00 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:51.204200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779e0ba5-c384-4e75-b7e5-8c1b69db1053 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:51.239294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:51.247523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b7caaa-b449-4094-a955-b45c16bc484c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:51.248131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b7caaa-b449-4094-a955-b45c16bc484c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.248410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b7caaa-b449-4094-a955-b45c16bc484c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.248684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b7caaa-b449-4094-a955-b45c16bc484c None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.249323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b7caaa-b449-4094-a955-b45c16bc484c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:51.249624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b7caaa-b449-4094-a955-b45c16bc484c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:51.251020 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b7caaa-b449-4094-a955-b45c16bc484c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:51.266202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 251/499] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:51.268013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b7caaa-b449-4094-a955-b45c16bc484c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:51.287564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8bfc2e-e2d2-4d98-a72f-48b15576cc00 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:51.293382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:51.323291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e34ff9-af8f-4071-b955-a3c132d48c67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:51.324014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e34ff9-af8f-4071-b955-a3c132d48c67 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.324367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e34ff9-af8f-4071-b955-a3c132d48c67 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.324650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e34ff9-af8f-4071-b955-a3c132d48c67 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.325307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e34ff9-af8f-4071-b955-a3c132d48c67 None admin] RBAC: Adding query filter params (name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:51.325609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e34ff9-af8f-4071-b955-a3c132d48c67 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:51.327001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e34ff9-af8f-4071-b955-a3c132d48c67 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:51.334982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 249/500] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:41:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:51.336732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91e3a3b-66d9-4056-a151-8c67fb518d47 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.337012 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91e3a3b-66d9-4056-a151-8c67fb518d47 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.337283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91e3a3b-66d9-4056-a151-8c67fb518d47 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.342693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 250/501] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:51.342693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91e3a3b-66d9-4056-a151-8c67fb518d47 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:51.388469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b7caaa-b449-4094-a955-b45c16bc484c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:51.425892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bc8cc0-4010-4bd3-a0c0-afedf85e3d47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:51.426522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bc8cc0-4010-4bd3-a0c0-afedf85e3d47 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.426797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bc8cc0-4010-4bd3-a0c0-afedf85e3d47 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.427070 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bc8cc0-4010-4bd3-a0c0-afedf85e3d47 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.444162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91e3a3b-66d9-4056-a151-8c67fb518d47 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:51.466184 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:51.478671 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed76e07d-c686-475e-84af-650fedb31186 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:51.479285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed76e07d-c686-475e-84af-650fedb31186 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.479565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed76e07d-c686-475e-84af-650fedb31186 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.479839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed76e07d-c686-475e-84af-650fedb31186 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.487261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed76e07d-c686-475e-84af-650fedb31186 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:51.488692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed76e07d-c686-475e-84af-650fedb31186 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:51.494661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bc8cc0-4010-4bd3-a0c0-afedf85e3d47 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:51.496115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bc8cc0-4010-4bd3-a0c0-afedf85e3d47 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-ed76e07d-c686-475e-84af-650fedb31186 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:41:51.499623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:51.500493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:51.501380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 251/502] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:51 2022] GET /identity/v3/users/glance => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:51.502495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed76e07d-c686-475e-84af-650fedb31186 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:51.522645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:51.595420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 252/503] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:51.596792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a9aef-f70f-4639-abed-9e14658f0138 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.597079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a9aef-f70f-4639-abed-9e14658f0138 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.597350 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40a9aef-f70f-4639-abed-9e14658f0138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.605461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed76e07d-c686-475e-84af-650fedb31186 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:51.615511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f40a9aef-f70f-4639-abed-9e14658f0138 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:51.616712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f40a9aef-f70f-4639-abed-9e14658f0138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:51.638009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf05560e-ebc3-495a-9875-1eb320dae448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:51.638657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf05560e-ebc3-495a-9875-1eb320dae448 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.638932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf05560e-ebc3-495a-9875-1eb320dae448 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.639258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf05560e-ebc3-495a-9875-1eb320dae448 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.650077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf05560e-ebc3-495a-9875-1eb320dae448 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:51.651506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf05560e-ebc3-495a-9875-1eb320dae448 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-bf05560e-ebc3-495a-9875-1eb320dae448 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:51.665098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:51.665977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:51.667053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:51.667053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:51.667053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:51.667053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 252/504] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:51.667053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf05560e-ebc3-495a-9875-1eb320dae448 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:51.685796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 253/505] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:51.686675 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40a9aef-f70f-4639-abed-9e14658f0138 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:51.705678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:51.748743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:51.776671 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf05560e-ebc3-495a-9875-1eb320dae448 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:51.793971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40a9aef-f70f-4639-abed-9e14658f0138 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:51.813984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fc1a54-4817-4875-be87-c05948e8bd56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:51.814633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc1a54-4817-4875-be87-c05948e8bd56 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.814912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc1a54-4817-4875-be87-c05948e8bd56 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.815184 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fc1a54-4817-4875-be87-c05948e8bd56 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.825944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fc1a54-4817-4875-be87-c05948e8bd56 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:51.829407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af02c6b-b2f5-4485-9c14-b3bce1902503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:51.830023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af02c6b-b2f5-4485-9c14-b3bce1902503 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.830340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af02c6b-b2f5-4485-9c14-b3bce1902503 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.830608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af02c6b-b2f5-4485-9c14-b3bce1902503 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.831322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af02c6b-b2f5-4485-9c14-b3bce1902503 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-21fc1a54-4817-4875-be87-c05948e8bd56 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:51.831528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:51.832482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:51.833317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:51.833950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af02c6b-b2f5-4485-9c14-b3bce1902503 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:51.834024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 253/506] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:41:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:51.834024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fc1a54-4817-4875-be87-c05948e8bd56 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:51.838253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af02c6b-b2f5-4485-9c14-b3bce1902503 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:51.855906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 254/507] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Oct 2 00:41:51 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:51.859251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af02c6b-b2f5-4485-9c14-b3bce1902503 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:51.932017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:51.939656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fc1a54-4817-4875-be87-c05948e8bd56 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:51.967831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af02c6b-b2f5-4485-9c14-b3bce1902503 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:51.973656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71ebec2-be97-466e-aa14-b50c26f9a5f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:51.974310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71ebec2-be97-466e-aa14-b50c26f9a5f8 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:51.974590 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71ebec2-be97-466e-aa14-b50c26f9a5f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:51.974866 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71ebec2-be97-466e-aa14-b50c26f9a5f8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:51.981011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:51.985756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71ebec2-be97-466e-aa14-b50c26f9a5f8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:51.987239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71ebec2-be97-466e-aa14-b50c26f9a5f8 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-e71ebec2-be97-466e-aa14-b50c26f9a5f8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:52.001502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:52.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:52.003421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:52.003421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:52.003421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:52.003593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34249b67-e58e-4ddb-b6b4-93f34b2924c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.003593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34249b67-e58e-4ddb-b6b4-93f34b2924c3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.003734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 254/508] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:52.003799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34249b67-e58e-4ddb-b6b4-93f34b2924c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.003799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34249b67-e58e-4ddb-b6b4-93f34b2924c3 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.003925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71ebec2-be97-466e-aa14-b50c26f9a5f8 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.003993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34249b67-e58e-4ddb-b6b4-93f34b2924c3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:52.003993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34249b67-e58e-4ddb-b6b4-93f34b2924c3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.005134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34249b67-e58e-4ddb-b6b4-93f34b2924c3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.017110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 255/509] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:52.017895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34249b67-e58e-4ddb-b6b4-93f34b2924c3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.108656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71ebec2-be97-466e-aa14-b50c26f9a5f8 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.116348 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34249b67-e58e-4ddb-b6b4-93f34b2924c3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.142353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc78d1d4-6d5a-45e9-98d3-3818b7bb3a4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.142942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc78d1d4-6d5a-45e9-98d3-3818b7bb3a4d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.143214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc78d1d4-6d5a-45e9-98d3-3818b7bb3a4d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.143479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc78d1d4-6d5a-45e9-98d3-3818b7bb3a4d None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.144124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc78d1d4-6d5a-45e9-98d3-3818b7bb3a4d None admin] RBAC: Adding query filter params (name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:52.144423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc78d1d4-6d5a-45e9-98d3-3818b7bb3a4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.145766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc78d1d4-6d5a-45e9-98d3-3818b7bb3a4d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.149674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f3c393-3e0e-4b20-ac3f-40f07816f344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.150302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3c393-3e0e-4b20-ac3f-40f07816f344 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.150582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3c393-3e0e-4b20-ac3f-40f07816f344 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.150854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f3c393-3e0e-4b20-ac3f-40f07816f344 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.153419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 255/510] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:52.154222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc78d1d4-6d5a-45e9-98d3-3818b7bb3a4d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.165912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:52.191538 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:52.217887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f3c393-3e0e-4b20-ac3f-40f07816f344 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.219381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f3c393-3e0e-4b20-ac3f-40f07816f344 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.254665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc78d1d4-6d5a-45e9-98d3-3818b7bb3a4d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.287436 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba3d032-08c0-4f59-8078-e39edb1dc2a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.288045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba3d032-08c0-4f59-8078-e39edb1dc2a2 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.288321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba3d032-08c0-4f59-8078-e39edb1dc2a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.288592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba3d032-08c0-4f59-8078-e39edb1dc2a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.298908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba3d032-08c0-4f59-8078-e39edb1dc2a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.300479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba3d032-08c0-4f59-8078-e39edb1dc2a2 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.314481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 256/511] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-4ba3d032-08c0-4f59-8078-e39edb1dc2a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:52.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:52.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:52.316779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:52.316779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:52.316779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:52.316952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f3c393-3e0e-4b20-ac3f-40f07816f344 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.317020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 256/512] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:52.317292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba3d032-08c0-4f59-8078-e39edb1dc2a2 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.375049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:52.383731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:52.413021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f3c393-3e0e-4b20-ac3f-40f07816f344 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.414524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba3d032-08c0-4f59-8078-e39edb1dc2a2 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.445766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be5690f-f0c9-44f0-b2fc-9c5220906769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.446398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be5690f-f0c9-44f0-b2fc-9c5220906769 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.446676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be5690f-f0c9-44f0-b2fc-9c5220906769 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.446946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be5690f-f0c9-44f0-b2fc-9c5220906769 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.447039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b4f432-a937-494b-a6eb-724183502ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.447560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b4f432-a937-494b-a6eb-724183502ce9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.447648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be5690f-f0c9-44f0-b2fc-9c5220906769 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:52.447838 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b4f432-a937-494b-a6eb-724183502ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.447935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be5690f-f0c9-44f0-b2fc-9c5220906769 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.448109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b4f432-a937-494b-a6eb-724183502ce9 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.449217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be5690f-f0c9-44f0-b2fc-9c5220906769 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.455376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b4f432-a937-494b-a6eb-724183502ce9 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.456791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b4f432-a937-494b-a6eb-724183502ce9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.461074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 257/513] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:52.461856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be5690f-f0c9-44f0-b2fc-9c5220906769 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-23b4f432-a937-494b-a6eb-724183502ce9 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:41:52.467925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:52.468954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 257/514] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/users/cinder => generated 84 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:52.469836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b4f432-a937-494b-a6eb-724183502ce9 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.560721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be5690f-f0c9-44f0-b2fc-9c5220906769 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.571972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b4f432-a937-494b-a6eb-724183502ce9 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.588977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:52.591334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:52.594190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5299ea67-8e4a-4102-9368-543fced03753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.594806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5299ea67-8e4a-4102-9368-543fced03753 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.595079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5299ea67-8e4a-4102-9368-543fced03753 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.595347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5299ea67-8e4a-4102-9368-543fced03753 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.605368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5299ea67-8e4a-4102-9368-543fced03753 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.606137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343b947e-adf5-4e09-8c21-d29f24eece22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.606753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343b947e-adf5-4e09-8c21-d29f24eece22 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.606847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5299ea67-8e4a-4102-9368-543fced03753 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.607030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343b947e-adf5-4e09-8c21-d29f24eece22 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.610118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343b947e-adf5-4e09-8c21-d29f24eece22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5299ea67-8e4a-4102-9368-543fced03753 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:52.621967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:52.622958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:52.623834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:52.623834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:52.623834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:52.623994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343b947e-adf5-4e09-8c21-d29f24eece22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.624062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 258/515] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:52.624134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343b947e-adf5-4e09-8c21-d29f24eece22 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.624209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5299ea67-8e4a-4102-9368-543fced03753 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-343b947e-adf5-4e09-8c21-d29f24eece22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:52.640492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:52.641435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:52.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:52.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:52.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:52.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 258/516] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:52.642586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343b947e-adf5-4e09-8c21-d29f24eece22 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.733740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5299ea67-8e4a-4102-9368-543fced03753 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.740510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343b947e-adf5-4e09-8c21-d29f24eece22 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.766637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64177b6b-a001-4222-aa6a-05a9acaa9335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.767246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64177b6b-a001-4222-aa6a-05a9acaa9335 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.767518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64177b6b-a001-4222-aa6a-05a9acaa9335 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.767781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64177b6b-a001-4222-aa6a-05a9acaa9335 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.768409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64177b6b-a001-4222-aa6a-05a9acaa9335 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:52.768714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64177b6b-a001-4222-aa6a-05a9acaa9335 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.770111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64177b6b-a001-4222-aa6a-05a9acaa9335 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.774898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93e41c5-140b-4863-a821-61e84bda7171 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.775522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93e41c5-140b-4863-a821-61e84bda7171 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.775803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93e41c5-140b-4863-a821-61e84bda7171 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.776072 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93e41c5-140b-4863-a821-61e84bda7171 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.782821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93e41c5-140b-4863-a821-61e84bda7171 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.783569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 259/517] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:52.784250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d93e41c5-140b-4863-a821-61e84bda7171 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.784351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64177b6b-a001-4222-aa6a-05a9acaa9335 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-d93e41c5-140b-4863-a821-61e84bda7171 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:41:52.796904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:52.797779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:52.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:41:52.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:41:52.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:41:52.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:41:52.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:41:52.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:41:52.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:52.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:52.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 259/518] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:52.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93e41c5-140b-4863-a821-61e84bda7171 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.828742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:52.833329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:52.885950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64177b6b-a001-4222-aa6a-05a9acaa9335 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.896505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d93e41c5-140b-4863-a821-61e84bda7171 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:52.918330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dadbfe4-4ce9-48a5-b401-03bdb1037426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.918932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dadbfe4-4ce9-48a5-b401-03bdb1037426 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.919211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dadbfe4-4ce9-48a5-b401-03bdb1037426 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.919480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dadbfe4-4ce9-48a5-b401-03bdb1037426 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.920196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dadbfe4-4ce9-48a5-b401-03bdb1037426 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:52.920501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dadbfe4-4ce9-48a5-b401-03bdb1037426 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.921929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dadbfe4-4ce9-48a5-b401-03bdb1037426 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.931951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5670165-0cdf-4d0a-8073-c9ff80198114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:52.932546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5670165-0cdf-4d0a-8073-c9ff80198114 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:52.932826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5670165-0cdf-4d0a-8073-c9ff80198114 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:52.933101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5670165-0cdf-4d0a-8073-c9ff80198114 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:52.933736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5670165-0cdf-4d0a-8073-c9ff80198114 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:52.934040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5670165-0cdf-4d0a-8073-c9ff80198114 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:52.935427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5670165-0cdf-4d0a-8073-c9ff80198114 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:52.938767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 260/519] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:52.939555 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dadbfe4-4ce9-48a5-b401-03bdb1037426 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:52.947955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 260/520] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:52.948731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5670165-0cdf-4d0a-8073-c9ff80198114 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:53.038592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:53.042115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dadbfe4-4ce9-48a5-b401-03bdb1037426 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:53.043655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:53.048126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5670165-0cdf-4d0a-8073-c9ff80198114 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:53.076089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7225f-5b33-47e6-b57b-e71ee1609364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:53.076689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7225f-5b33-47e6-b57b-e71ee1609364 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:53.076963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7225f-5b33-47e6-b57b-e71ee1609364 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:53.077236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7225f-5b33-47e6-b57b-e71ee1609364 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:53.077859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc7225f-5b33-47e6-b57b-e71ee1609364 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:53.078189 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc7225f-5b33-47e6-b57b-e71ee1609364 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:53.079606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc7225f-5b33-47e6-b57b-e71ee1609364 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:53.081674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58249ff4-ddce-4fad-9b72-b5a41d77d30b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:53.082305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58249ff4-ddce-4fad-9b72-b5a41d77d30b None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:53.082585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58249ff4-ddce-4fad-9b72-b5a41d77d30b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:53.082855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58249ff4-ddce-4fad-9b72-b5a41d77d30b None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:53.083487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58249ff4-ddce-4fad-9b72-b5a41d77d30b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:53.084895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58249ff4-ddce-4fad-9b72-b5a41d77d30b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:53.092296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 261/521] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:53.093093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7225f-5b33-47e6-b57b-e71ee1609364 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:53.107245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 261/522] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:41:52 2022] POST /identity/v3/users => generated 247 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:41:53.117968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58249ff4-ddce-4fad-9b72-b5a41d77d30b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:53.215394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7225f-5b33-47e6-b57b-e71ee1609364 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:53.223212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58249ff4-ddce-4fad-9b72-b5a41d77d30b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:53.255176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ba87f1-e90d-418f-b9fe-5434bf0d74d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:53.255791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ba87f1-e90d-418f-b9fe-5434bf0d74d3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:53.256064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ba87f1-e90d-418f-b9fe-5434bf0d74d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:53.256326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ba87f1-e90d-418f-b9fe-5434bf0d74d3 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:53.257030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ba87f1-e90d-418f-b9fe-5434bf0d74d3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:53.257333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ba87f1-e90d-418f-b9fe-5434bf0d74d3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:53.259625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ba87f1-e90d-418f-b9fe-5434bf0d74d3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:53.263428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bafbd8a-fa36-46b4-8f10-b4f60d1611ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:53.264292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bafbd8a-fa36-46b4-8f10-b4f60d1611ea None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:53.264706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bafbd8a-fa36-46b4-8f10-b4f60d1611ea None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:53.265116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bafbd8a-fa36-46b4-8f10-b4f60d1611ea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:53.278668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 262/523] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:41:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:53.278950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ba87f1-e90d-418f-b9fe-5434bf0d74d3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:53.290593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:53.309794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:53.351857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bafbd8a-fa36-46b4-8f10-b4f60d1611ea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:53.353685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bafbd8a-fa36-46b4-8f10-b4f60d1611ea None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:53.391124 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ba87f1-e90d-418f-b9fe-5434bf0d74d3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:53.425610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c191b8-6b06-4ab3-a43f-77beaf33a5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:53.426265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c191b8-6b06-4ab3-a43f-77beaf33a5ec None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:53.426517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c191b8-6b06-4ab3-a43f-77beaf33a5ec None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:53.426783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c191b8-6b06-4ab3-a43f-77beaf33a5ec None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:53.433897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c191b8-6b06-4ab3-a43f-77beaf33a5ec None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:53.435403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c191b8-6b06-4ab3-a43f-77beaf33a5ec None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f0c191b8-6b06-4ab3-a43f-77beaf33a5ec None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:41:53.449416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:53.450366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:53.451247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 263/524] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:41:53 2022] GET /identity/v3/users/neutron => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:53.451887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c191b8-6b06-4ab3-a43f-77beaf33a5ec None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:53.466333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 262/525] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:53.467176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bafbd8a-fa36-46b4-8f10-b4f60d1611ea None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:53.518207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:53.546781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:53.567098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c191b8-6b06-4ab3-a43f-77beaf33a5ec None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:53.568625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bafbd8a-fa36-46b4-8f10-b4f60d1611ea None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:53.601950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd12052-6a8b-4257-a027-9fe9815cf046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:53.602259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995c553d-a654-4a70-85be-af7157637123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:53.602495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd12052-6a8b-4257-a027-9fe9815cf046 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:53.602768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd12052-6a8b-4257-a027-9fe9815cf046 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:53.602873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995c553d-a654-4a70-85be-af7157637123 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:53.603036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd12052-6a8b-4257-a027-9fe9815cf046 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:53.603126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995c553d-a654-4a70-85be-af7157637123 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:53.603374 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995c553d-a654-4a70-85be-af7157637123 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:53.603745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd12052-6a8b-4257-a027-9fe9815cf046 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=dad93d99b865407aa120035015f4692a) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:53.604049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd12052-6a8b-4257-a027-9fe9815cf046 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:53.605461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd12052-6a8b-4257-a027-9fe9815cf046 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:53.610376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995c553d-a654-4a70-85be-af7157637123 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:53.618423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 263/526] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:53 2022] GET /identity/v3/role_assignments?user.id=dad93d99b865407aa120035015f4692a&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:53.618423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd12052-6a8b-4257-a027-9fe9815cf046 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:53.618741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995c553d-a654-4a70-85be-af7157637123 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-995c553d-a654-4a70-85be-af7157637123 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:41:53.631203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:53.632114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:53.633159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:41:53.633159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:41:53.633159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:41:53.633159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:41:53.633159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:41:53.633159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:41:53.633159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:53.633159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:53.633159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 264/527] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:53.633884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995c553d-a654-4a70-85be-af7157637123 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:53.748769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd12052-6a8b-4257-a027-9fe9815cf046 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:53.767090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995c553d-a654-4a70-85be-af7157637123 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:53.768100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:53.798549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d91a4-cb95-41fa-beef-3d8848d8f6c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:53.799575 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d91a4-cb95-41fa-beef-3d8848d8f6c8 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:53.799982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d91a4-cb95-41fa-beef-3d8848d8f6c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:53.800377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d91a4-cb95-41fa-beef-3d8848d8f6c8 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:53.801115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d91a4-cb95-41fa-beef-3d8848d8f6c8 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:53.801417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d91a4-cb95-41fa-beef-3d8848d8f6c8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:53.802939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d91a4-cb95-41fa-beef-3d8848d8f6c8 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:53.809402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a46c6ae-1cc6-4bb2-bf40-113dbffa697e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:53.810040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a46c6ae-1cc6-4bb2-bf40-113dbffa697e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:53.810344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a46c6ae-1cc6-4bb2-bf40-113dbffa697e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:53.810617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a46c6ae-1cc6-4bb2-bf40-113dbffa697e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:53.821615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a46c6ae-1cc6-4bb2-bf40-113dbffa697e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:53.823117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a46c6ae-1cc6-4bb2-bf40-113dbffa697e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:53.831609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-9a46c6ae-1cc6-4bb2-bf40-113dbffa697e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:53.839413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:53.840322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:53.841174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:53.841174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:53.841174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:53.843397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 265/528] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:53.844551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 264/529] 10.222.0.25 () {56 vars in 1139 bytes} [Sun Oct 2 00:41:53 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:53.849135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a46c6ae-1cc6-4bb2-bf40-113dbffa697e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:53.851007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d91a4-cb95-41fa-beef-3d8848d8f6c8 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:53.958801 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a46c6ae-1cc6-4bb2-bf40-113dbffa697e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:53.979442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d91a4-cb95-41fa-beef-3d8848d8f6c8 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:54.000936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e959127b-9702-48d3-8b46-6461d7b78525 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:54.001943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e959127b-9702-48d3-8b46-6461d7b78525 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:54.002270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e959127b-9702-48d3-8b46-6461d7b78525 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:54.002543 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e959127b-9702-48d3-8b46-6461d7b78525 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:54.003229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e959127b-9702-48d3-8b46-6461d7b78525 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:54.003529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e959127b-9702-48d3-8b46-6461d7b78525 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:54.005385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e959127b-9702-48d3-8b46-6461d7b78525 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:54.019761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 266/530] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:41:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:54.020824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e959127b-9702-48d3-8b46-6461d7b78525 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:54.022961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b095ee-0322-46ef-a746-df4e369595d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:54.023888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b095ee-0322-46ef-a746-df4e369595d7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:54.024484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b095ee-0322-46ef-a746-df4e369595d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:54.024765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b095ee-0322-46ef-a746-df4e369595d7 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:54.028558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b095ee-0322-46ef-a746-df4e369595d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:54.028558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b095ee-0322-46ef-a746-df4e369595d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:54.028558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b095ee-0322-46ef-a746-df4e369595d7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:54.045869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 265/531] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:54.047226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b095ee-0322-46ef-a746-df4e369595d7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:54.089699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:54.099894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:54.163778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e959127b-9702-48d3-8b46-6461d7b78525 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:54.170683 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b095ee-0322-46ef-a746-df4e369595d7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:54.204631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e5058b-a97f-45ba-b6bf-9ae80f0e05dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:54.205323 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5058b-a97f-45ba-b6bf-9ae80f0e05dd None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:54.205686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5058b-a97f-45ba-b6bf-9ae80f0e05dd None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:54.206038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5058b-a97f-45ba-b6bf-9ae80f0e05dd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:54.206903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5058b-a97f-45ba-b6bf-9ae80f0e05dd None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=30fc59f80f544cef8b3869e8d37bb341) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:54.207051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc93ea44-8797-4cc9-81f9-6d1872ad021f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:54.207218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5058b-a97f-45ba-b6bf-9ae80f0e05dd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:54.207744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc93ea44-8797-4cc9-81f9-6d1872ad021f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:54.208080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc93ea44-8797-4cc9-81f9-6d1872ad021f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:54.208410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc93ea44-8797-4cc9-81f9-6d1872ad021f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:54.208683 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5058b-a97f-45ba-b6bf-9ae80f0e05dd None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:54.214626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 266/532] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:54 2022] GET /identity/v3/role_assignments?user.id=30fc59f80f544cef8b3869e8d37bb341&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:54.215356 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e5058b-a97f-45ba-b6bf-9ae80f0e05dd None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:54.357900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc93ea44-8797-4cc9-81f9-6d1872ad021f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:54.359426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc93ea44-8797-4cc9-81f9-6d1872ad021f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:54.418652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:54.430879 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e5058b-a97f-45ba-b6bf-9ae80f0e05dd None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:54.479220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:54.506921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1e854d-b4e5-4c8a-86a3-cc3d8be636fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:54.507651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1e854d-b4e5-4c8a-86a3-cc3d8be636fd None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:54.507931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1e854d-b4e5-4c8a-86a3-cc3d8be636fd None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:54.508202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1e854d-b4e5-4c8a-86a3-cc3d8be636fd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:54.521449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1e854d-b4e5-4c8a-86a3-cc3d8be636fd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:54.523092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1e854d-b4e5-4c8a-86a3-cc3d8be636fd None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-6e1e854d-b4e5-4c8a-86a3-cc3d8be636fd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:41:54.551284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:54.553326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:54.555658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:41:54.555658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:41:54.555658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:41:54.555658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:41:54.555658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:41:54.555658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:41:54.555658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:54.555658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:54.555658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 267/533] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:54.556238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 267/534] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:54.558727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1e854d-b4e5-4c8a-86a3-cc3d8be636fd None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:54.561781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc93ea44-8797-4cc9-81f9-6d1872ad021f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:54.685067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1e854d-b4e5-4c8a-86a3-cc3d8be636fd None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:54.745393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc93ea44-8797-4cc9-81f9-6d1872ad021f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:54.748162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f4dc8f-569f-4911-9990-81d86396de68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:54.748769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f4dc8f-569f-4911-9990-81d86396de68 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:54.749041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f4dc8f-569f-4911-9990-81d86396de68 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:54.749308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f4dc8f-569f-4911-9990-81d86396de68 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:54.749992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f4dc8f-569f-4911-9990-81d86396de68 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:54.750316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f4dc8f-569f-4911-9990-81d86396de68 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:54.751744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f4dc8f-569f-4911-9990-81d86396de68 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:54.773812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 268/535] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:41:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:54.782715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f4dc8f-569f-4911-9990-81d86396de68 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:54.810731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:54.815334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b4ebbc-a193-4e0a-908c-fb60e869b8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:54.815987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b4ebbc-a193-4e0a-908c-fb60e869b8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:54.816263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b4ebbc-a193-4e0a-908c-fb60e869b8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:54.816532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b4ebbc-a193-4e0a-908c-fb60e869b8d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:54.838865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b4ebbc-a193-4e0a-908c-fb60e869b8d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:54.840328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b4ebbc-a193-4e0a-908c-fb60e869b8d7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-48b4ebbc-a193-4e0a-908c-fb60e869b8d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:54.869386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:54.871601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:54.873661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:54.873661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:54.873661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:54.873661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 268/536] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:54.879761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b4ebbc-a193-4e0a-908c-fb60e869b8d7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:54.894608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:54.968966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f4dc8f-569f-4911-9990-81d86396de68 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:55.020242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39706973-6007-4530-a31c-367af6e1c269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:55.020766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39706973-6007-4530-a31c-367af6e1c269 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:55.021040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39706973-6007-4530-a31c-367af6e1c269 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:55.021308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39706973-6007-4530-a31c-367af6e1c269 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:55.022027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39706973-6007-4530-a31c-367af6e1c269 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=eb325d3c277b45c2a5c4e0e94078be29) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:55.022354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39706973-6007-4530-a31c-367af6e1c269 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:55.023864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39706973-6007-4530-a31c-367af6e1c269 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:55.044766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 269/537] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:54 2022] GET /identity/v3/role_assignments?user.id=eb325d3c277b45c2a5c4e0e94078be29&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 259 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:55.077690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b4ebbc-a193-4e0a-908c-fb60e869b8d7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:55.124864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b6f7e1-3230-4623-9a77-d732e83afe34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:55.125477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b6f7e1-3230-4623-9a77-d732e83afe34 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:55.125748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b6f7e1-3230-4623-9a77-d732e83afe34 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:55.126016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b6f7e1-3230-4623-9a77-d732e83afe34 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:55.126673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b6f7e1-3230-4623-9a77-d732e83afe34 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:55.126967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b6f7e1-3230-4623-9a77-d732e83afe34 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:55.128329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b6f7e1-3230-4623-9a77-d732e83afe34 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:55.145302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 269/538] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:55.151480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39706973-6007-4530-a31c-367af6e1c269 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:55.267452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:55.298586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39706973-6007-4530-a31c-367af6e1c269 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:55.332939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafcc048-7408-45a0-9527-f9d210123c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:55.333566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafcc048-7408-45a0-9527-f9d210123c0a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:55.333845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafcc048-7408-45a0-9527-f9d210123c0a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:55.334149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bafcc048-7408-45a0-9527-f9d210123c0a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:55.341401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafcc048-7408-45a0-9527-f9d210123c0a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:55.342884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bafcc048-7408-45a0-9527-f9d210123c0a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-bafcc048-7408-45a0-9527-f9d210123c0a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:41:55.354172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:55.354989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:55.355824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:41:55.355824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:41:55.355824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:41:55.355824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:41:55.355824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:41:55.355824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:41:55.355824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:55.355824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:55.355824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 270/539] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:55.362697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b6f7e1-3230-4623-9a77-d732e83afe34 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:55.468305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:55.526449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b6f7e1-3230-4623-9a77-d732e83afe34 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:55.569080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428e2c7a-2cf1-46af-b7aa-11eff3f28d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:55.569606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e2c7a-2cf1-46af-b7aa-11eff3f28d1b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:55.569882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e2c7a-2cf1-46af-b7aa-11eff3f28d1b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:55.570706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e2c7a-2cf1-46af-b7aa-11eff3f28d1b None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:55.571392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428e2c7a-2cf1-46af-b7aa-11eff3f28d1b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:55.571694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428e2c7a-2cf1-46af-b7aa-11eff3f28d1b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:55.573131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428e2c7a-2cf1-46af-b7aa-11eff3f28d1b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:55.589112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 270/540] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:41:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:55.594852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafcc048-7408-45a0-9527-f9d210123c0a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:55.639252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eafe815-1035-407d-ac94-ed22913d7a72 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:55.640782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eafe815-1035-407d-ac94-ed22913d7a72 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:55.641644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eafe815-1035-407d-ac94-ed22913d7a72 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:55.644378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 271/541] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:55 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:55.653404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0240e99a-a0f1-4d47-bfe6-517522357f28 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:55.654235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0240e99a-a0f1-4d47-bfe6-517522357f28 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:55.655074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0240e99a-a0f1-4d47-bfe6-517522357f28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:55.690097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0240e99a-a0f1-4d47-bfe6-517522357f28 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:55.691318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0240e99a-a0f1-4d47-bfe6-517522357f28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:55.747418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bafcc048-7408-45a0-9527-f9d210123c0a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:55.796719 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 272/542] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:41:55.807096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9113a70e-039e-4f41-b938-3c00060f91c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:55.807812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9113a70e-039e-4f41-b938-3c00060f91c7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:55.808180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9113a70e-039e-4f41-b938-3c00060f91c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:55.808546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9113a70e-039e-4f41-b938-3c00060f91c7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:55.809362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9113a70e-039e-4f41-b938-3c00060f91c7 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=e75811ee9ed7490985fed336ab9feb19) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:55.809786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9113a70e-039e-4f41-b938-3c00060f91c7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:55.811417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9113a70e-039e-4f41-b938-3c00060f91c7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:55.817351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 271/543] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:55 2022] GET /identity/v3/role_assignments?user.id=e75811ee9ed7490985fed336ab9feb19&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 259 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:55.992279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e6844c-4f5b-4668-b369-72cfac19fec2 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:55.992644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e6844c-4f5b-4668-b369-72cfac19fec2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:55.992845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e6844c-4f5b-4668-b369-72cfac19fec2 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:55.994623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 273/544] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:56.000395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0f4b7e-74f7-4eb0-be1c-c52d563aeefd None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:56.000679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0f4b7e-74f7-4eb0-be1c-c52d563aeefd None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:56.000946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0f4b7e-74f7-4eb0-be1c-c52d563aeefd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:56.007509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:56.028858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8e0f4b7e-74f7-4eb0-be1c-c52d563aeefd None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:56.029741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8e0f4b7e-74f7-4eb0-be1c-c52d563aeefd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:56.143901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 272/545] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:56.254876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e6844c-4f5b-4668-b369-72cfac19fec2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:56.323464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:56.381774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e6844c-4f5b-4668-b369-72cfac19fec2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:56.421384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94546c2-5056-477a-aee7-0dfb93a89155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:56.422006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94546c2-5056-477a-aee7-0dfb93a89155 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:56.422298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94546c2-5056-477a-aee7-0dfb93a89155 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:56.422562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94546c2-5056-477a-aee7-0dfb93a89155 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:56.433085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94546c2-5056-477a-aee7-0dfb93a89155 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b94546c2-5056-477a-aee7-0dfb93a89155 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:56.437926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:56.438836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:56.439634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 274/546] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:41:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:56.447247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0f4b7e-74f7-4eb0-be1c-c52d563aeefd None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:56.535680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94546c2-5056-477a-aee7-0dfb93a89155 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:56.590386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0f4b7e-74f7-4eb0-be1c-c52d563aeefd None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:56.647966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:56.658542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e9c89e-ca63-4aff-b7c6-6ecc77cb76e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:56.659182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e9c89e-ca63-4aff-b7c6-6ecc77cb76e7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:56.659464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e9c89e-ca63-4aff-b7c6-6ecc77cb76e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:56.659737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e9c89e-ca63-4aff-b7c6-6ecc77cb76e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:56.660406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e9c89e-ca63-4aff-b7c6-6ecc77cb76e7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:56.660710 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e9c89e-ca63-4aff-b7c6-6ecc77cb76e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:56.662107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e9c89e-ca63-4aff-b7c6-6ecc77cb76e7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:56.683652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 273/547] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:41:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:41:56.687836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e9c89e-ca63-4aff-b7c6-6ecc77cb76e7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:56.716500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94546c2-5056-477a-aee7-0dfb93a89155 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:56.744170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:56.765141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac08647e-9ce1-4d17-a765-d86185a393df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:56.766172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac08647e-9ce1-4d17-a765-d86185a393df None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:56.766800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac08647e-9ce1-4d17-a765-d86185a393df None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:56.767265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac08647e-9ce1-4d17-a765-d86185a393df None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:56.778818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac08647e-9ce1-4d17-a765-d86185a393df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ac08647e-9ce1-4d17-a765-d86185a393df None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:56.784129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:56.785005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:56.785844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:56.788685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 275/548] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:41:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:56.790155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979500a4-899a-4b43-ba76-cdd1b80250b4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:56.790646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979500a4-899a-4b43-ba76-cdd1b80250b4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:56.791121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979500a4-899a-4b43-ba76-cdd1b80250b4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:56.792703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 276/549] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:56.793999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979500a4-899a-4b43-ba76-cdd1b80250b4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:56.805189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e9c89e-ca63-4aff-b7c6-6ecc77cb76e7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:56.842652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7791fa92-7043-46cd-ba6a-394025847515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:56.843306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7791fa92-7043-46cd-ba6a-394025847515 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:56.843606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7791fa92-7043-46cd-ba6a-394025847515 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:56.843975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7791fa92-7043-46cd-ba6a-394025847515 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:56.855514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7791fa92-7043-46cd-ba6a-394025847515 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:56.857083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7791fa92-7043-46cd-ba6a-394025847515 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-7791fa92-7043-46cd-ba6a-394025847515 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:56.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:56.874656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:56.875602 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:56.875602 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:56.875602 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:56.879026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 274/550] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:56.880337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f7b238-b6aa-4dc6-97a3-6af2d847cb8f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:56.880627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f7b238-b6aa-4dc6-97a3-6af2d847cb8f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:56.880902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f7b238-b6aa-4dc6-97a3-6af2d847cb8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:56.901151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e3f7b238-b6aa-4dc6-97a3-6af2d847cb8f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:56.902378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e3f7b238-b6aa-4dc6-97a3-6af2d847cb8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:56.923164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979500a4-899a-4b43-ba76-cdd1b80250b4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:56.945873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:56.968501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17503964-a771-4f3d-be4c-8e0eb1000c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:56.969980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17503964-a771-4f3d-be4c-8e0eb1000c8c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:56.970786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17503964-a771-4f3d-be4c-8e0eb1000c8c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:56.971295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17503964-a771-4f3d-be4c-8e0eb1000c8c None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:56.972159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17503964-a771-4f3d-be4c-8e0eb1000c8c None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:56.972656 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17503964-a771-4f3d-be4c-8e0eb1000c8c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:56.974323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17503964-a771-4f3d-be4c-8e0eb1000c8c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:56.974501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 275/551] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:56.975406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f7b238-b6aa-4dc6-97a3-6af2d847cb8f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:56.985748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 277/552] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:41:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:56.990901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17503964-a771-4f3d-be4c-8e0eb1000c8c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:57.055469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:57.090740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f7b238-b6aa-4dc6-97a3-6af2d847cb8f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:57.122852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17503964-a771-4f3d-be4c-8e0eb1000c8c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:57.138277 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e05e811-470d-4a6c-b015-6d208e1e46e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:57.138897 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e05e811-470d-4a6c-b015-6d208e1e46e1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:57.139176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e05e811-470d-4a6c-b015-6d208e1e46e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:57.139445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e05e811-470d-4a6c-b015-6d208e1e46e1 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:57.140088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e05e811-470d-4a6c-b015-6d208e1e46e1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:57.140389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e05e811-470d-4a6c-b015-6d208e1e46e1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:57.144667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e05e811-470d-4a6c-b015-6d208e1e46e1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:57.162211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 276/553] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:57.162376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e05e811-470d-4a6c-b015-6d208e1e46e1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:57.182582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b9e688-ea8a-42c7-9862-09a155880009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:57.183621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b9e688-ea8a-42c7-9862-09a155880009 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:57.184106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b9e688-ea8a-42c7-9862-09a155880009 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:57.184572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b9e688-ea8a-42c7-9862-09a155880009 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:57.199937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:57.207830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b9e688-ea8a-42c7-9862-09a155880009 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:57.209540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b9e688-ea8a-42c7-9862-09a155880009 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-56b9e688-ea8a-42c7-9862-09a155880009 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:57.240187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:57.241262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:57.242262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:57.242262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:57.242262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:57.250513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 278/554] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:57.256139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b9e688-ea8a-42c7-9862-09a155880009 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:57.274125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e05e811-470d-4a6c-b015-6d208e1e46e1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:57.309666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dbe45d-74e2-4ff7-921e-df86b0e2828e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:57.310296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dbe45d-74e2-4ff7-921e-df86b0e2828e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:57.310708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dbe45d-74e2-4ff7-921e-df86b0e2828e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:57.311221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2dbe45d-74e2-4ff7-921e-df86b0e2828e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:57.319093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2dbe45d-74e2-4ff7-921e-df86b0e2828e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:57.320580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2dbe45d-74e2-4ff7-921e-df86b0e2828e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-b2dbe45d-74e2-4ff7-921e-df86b0e2828e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:41:57.334385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:57.335330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:57.336845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 277/555] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:41:57 2022] GET /identity/v3/users/swift => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:41:57.337409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dbe45d-74e2-4ff7-921e-df86b0e2828e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:57.419154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:57.432449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b9e688-ea8a-42c7-9862-09a155880009 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:57.448550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2dbe45d-74e2-4ff7-921e-df86b0e2828e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:57.450364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:57.471681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c85b5d-0054-4a56-b4e9-ada59dacdcf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:57.472281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c85b5d-0054-4a56-b4e9-ada59dacdcf5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:57.472552 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c85b5d-0054-4a56-b4e9-ada59dacdcf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:57.472918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c85b5d-0054-4a56-b4e9-ada59dacdcf5 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:57.473652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c85b5d-0054-4a56-b4e9-ada59dacdcf5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:57.473953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c85b5d-0054-4a56-b4e9-ada59dacdcf5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:57.475434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c85b5d-0054-4a56-b4e9-ada59dacdcf5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:57.484976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dbef9c-932c-44d5-92ff-91a3c94d3d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:57.485658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dbef9c-932c-44d5-92ff-91a3c94d3d36 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:57.485939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dbef9c-932c-44d5-92ff-91a3c94d3d36 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:57.486256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dbef9c-932c-44d5-92ff-91a3c94d3d36 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:57.488565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 279/556] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:57.489304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c85b5d-0054-4a56-b4e9-ada59dacdcf5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:57.498484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dbef9c-932c-44d5-92ff-91a3c94d3d36 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-81dbef9c-932c-44d5-92ff-91a3c94d3d36 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:57.503919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:57.506518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 278/557] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:41:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:57.507790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dbef9c-932c-44d5-92ff-91a3c94d3d36 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:57.645081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c85b5d-0054-4a56-b4e9-ada59dacdcf5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:57.684647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dbef9c-932c-44d5-92ff-91a3c94d3d36 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:57.701104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b989f5c3-4748-4880-a388-9bd2528483ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:57.701711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b989f5c3-4748-4880-a388-9bd2528483ef None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:57.701988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b989f5c3-4748-4880-a388-9bd2528483ef None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:57.702290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b989f5c3-4748-4880-a388-9bd2528483ef None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:57.703012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b989f5c3-4748-4880-a388-9bd2528483ef None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:57.703773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b989f5c3-4748-4880-a388-9bd2528483ef None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:57.704755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b989f5c3-4748-4880-a388-9bd2528483ef None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:57.726937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 280/558] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 2 00:41:57 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:57.728790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b989f5c3-4748-4880-a388-9bd2528483ef None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:57.739321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa2037e-6d94-4b07-9c2e-2ff771bf06db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:57.740196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa2037e-6d94-4b07-9c2e-2ff771bf06db None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:57.740656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa2037e-6d94-4b07-9c2e-2ff771bf06db None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:57.740792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:57.741179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa2037e-6d94-4b07-9c2e-2ff771bf06db None admin] PATH_INFO: `/v3/users/nova` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:57.751155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa2037e-6d94-4b07-9c2e-2ff771bf06db None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:57.753167 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa2037e-6d94-4b07-9c2e-2ff771bf06db None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-1aa2037e-6d94-4b07-9c2e-2ff771bf06db None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:41:57.767236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:57.768500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:57.769661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:57.769661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:57.769661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:41:57.769661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:41:57.769661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:41:57.769661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:41:57.769661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:41:57.769661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:41:57.769661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:41:57.769661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:57.775760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 279/559] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 2 00:41:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:57.776871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41b0224-3da5-492e-b8c8-f85cd24b1700 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:57.777231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41b0224-3da5-492e-b8c8-f85cd24b1700 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:57.777580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41b0224-3da5-492e-b8c8-f85cd24b1700 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:57.779063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 280/560] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:57.779816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41b0224-3da5-492e-b8c8-f85cd24b1700 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:57.799129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:57.857350 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b989f5c3-4748-4880-a388-9bd2528483ef None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:57.891784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f359275-b000-4a6c-8f31-a5e0811bbe89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:57.892678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f359275-b000-4a6c-8f31-a5e0811bbe89 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:57.893145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f359275-b000-4a6c-8f31-a5e0811bbe89 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:57.893523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f359275-b000-4a6c-8f31-a5e0811bbe89 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:57.894360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f359275-b000-4a6c-8f31-a5e0811bbe89 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:57.894767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f359275-b000-4a6c-8f31-a5e0811bbe89 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:57.896318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f359275-b000-4a6c-8f31-a5e0811bbe89 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:57.909451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 281/561] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:41:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:57.910494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f359275-b000-4a6c-8f31-a5e0811bbe89 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:57.921476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41b0224-3da5-492e-b8c8-f85cd24b1700 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:57.962199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b5b8ab-73c3-4b09-9c81-ac764f389c28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:57.962750 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b5b8ab-73c3-4b09-9c81-ac764f389c28 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:57.963024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b5b8ab-73c3-4b09-9c81-ac764f389c28 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:57.963295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b5b8ab-73c3-4b09-9c81-ac764f389c28 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:57.998587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:58.045326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b5b8ab-73c3-4b09-9c81-ac764f389c28 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:58.047116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b5b8ab-73c3-4b09-9c81-ac764f389c28 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:58.052834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f359275-b000-4a6c-8f31-a5e0811bbe89 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:58.067398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:58.090833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1080d7b6-4400-4b3f-9e60-b18f2fec7b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:58.091441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1080d7b6-4400-4b3f-9e60-b18f2fec7b78 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.091718 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1080d7b6-4400-4b3f-9e60-b18f2fec7b78 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.091992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1080d7b6-4400-4b3f-9e60-b18f2fec7b78 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.092711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1080d7b6-4400-4b3f-9e60-b18f2fec7b78 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:58.093038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1080d7b6-4400-4b3f-9e60-b18f2fec7b78 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:58.094474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1080d7b6-4400-4b3f-9e60-b18f2fec7b78 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:58.112705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 282/562] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 2 00:41:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:58.113791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5593ce25-6cb5-4523-98a3-3a800fadeac4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.114132 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5593ce25-6cb5-4523-98a3-3a800fadeac4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.114377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5593ce25-6cb5-4523-98a3-3a800fadeac4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.131962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5593ce25-6cb5-4523-98a3-3a800fadeac4 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:58.133179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5593ce25-6cb5-4523-98a3-3a800fadeac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:58.165722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 281/563] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:41:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:41:58.166553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b5b8ab-73c3-4b09-9c81-ac764f389c28 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:58.206190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 283/564] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:58.207462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5593ce25-6cb5-4523-98a3-3a800fadeac4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:58.286543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b5b8ab-73c3-4b09-9c81-ac764f389c28 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:58.291451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:58.308301 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:58.333261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacf3817-ffb3-4e6c-9aae-7a6cf3fd5f56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:58.334333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacf3817-ffb3-4e6c-9aae-7a6cf3fd5f56 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.334615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacf3817-ffb3-4e6c-9aae-7a6cf3fd5f56 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.334893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacf3817-ffb3-4e6c-9aae-7a6cf3fd5f56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.344046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5593ce25-6cb5-4523-98a3-3a800fadeac4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:58.349031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacf3817-ffb3-4e6c-9aae-7a6cf3fd5f56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:58.350527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacf3817-ffb3-4e6c-9aae-7a6cf3fd5f56 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-bacf3817-ffb3-4e6c-9aae-7a6cf3fd5f56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:58.369522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:58.370788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:58.371696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:58.371696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:58.371696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:58.373675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 282/565] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:58.374776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacf3817-ffb3-4e6c-9aae-7a6cf3fd5f56 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:58.386855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8321b36c-7962-40f1-9ac8-a01a82b88322 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:58.387488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8321b36c-7962-40f1-9ac8-a01a82b88322 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.387764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8321b36c-7962-40f1-9ac8-a01a82b88322 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.388040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8321b36c-7962-40f1-9ac8-a01a82b88322 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.402874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8321b36c-7962-40f1-9ac8-a01a82b88322 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:58.404655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8321b36c-7962-40f1-9ac8-a01a82b88322 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8321b36c-7962-40f1-9ac8-a01a82b88322 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:58.430780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:58.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:58.432683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:58.432683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:58.432683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:58.434154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 284/566] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:58.435325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8321b36c-7962-40f1-9ac8-a01a82b88322 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:58.509542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacf3817-ffb3-4e6c-9aae-7a6cf3fd5f56 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:58.543147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8321b36c-7962-40f1-9ac8-a01a82b88322 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:58.544875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ce9bd7-9724-4be4-b694-8b59735e3ecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:58.545510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ce9bd7-9724-4be4-b694-8b59735e3ecd None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.545791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ce9bd7-9724-4be4-b694-8b59735e3ecd None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.546078 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ce9bd7-9724-4be4-b694-8b59735e3ecd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.553270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:58.557108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ce9bd7-9724-4be4-b694-8b59735e3ecd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:58.558603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ce9bd7-9724-4be4-b694-8b59735e3ecd None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-69ce9bd7-9724-4be4-b694-8b59735e3ecd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:58.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:58.573938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:58.574862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:58.574862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:58.574862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:58.574862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 283/567] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:58.575105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ce9bd7-9724-4be4-b694-8b59735e3ecd None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:58.581256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8197a1-f91f-42bf-bf4f-2fd992e261cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:58.581860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8197a1-f91f-42bf-bf4f-2fd992e261cb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.582170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8197a1-f91f-42bf-bf4f-2fd992e261cb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.582445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8197a1-f91f-42bf-bf4f-2fd992e261cb None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.583082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8197a1-f91f-42bf-bf4f-2fd992e261cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:58.583385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8197a1-f91f-42bf-bf4f-2fd992e261cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:58.584749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8197a1-f91f-42bf-bf4f-2fd992e261cb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:58.596946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 285/568] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:58.597725 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8197a1-f91f-42bf-bf4f-2fd992e261cb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:58.602653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:58.674190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ce9bd7-9724-4be4-b694-8b59735e3ecd None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:58.696192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8197a1-f91f-42bf-bf4f-2fd992e261cb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:58.707625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b4dfb0-0d7e-4169-b89c-af51e81e3eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:58.708218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b4dfb0-0d7e-4169-b89c-af51e81e3eb0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.708502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b4dfb0-0d7e-4169-b89c-af51e81e3eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.708772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b4dfb0-0d7e-4169-b89c-af51e81e3eb0 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.709401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b4dfb0-0d7e-4169-b89c-af51e81e3eb0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:58.709699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b4dfb0-0d7e-4169-b89c-af51e81e3eb0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:58.711166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b4dfb0-0d7e-4169-b89c-af51e81e3eb0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:58.722424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 284/569] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:58.723564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eade2be5-38eb-410b-b9a2-062cf3b58a6d None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.723844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eade2be5-38eb-410b-b9a2-062cf3b58a6d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.724115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eade2be5-38eb-410b-b9a2-062cf3b58a6d None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.725291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 285/570] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:41:58 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:41:58.726004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eade2be5-38eb-410b-b9a2-062cf3b58a6d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:58.733512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa3561e-13b5-4e6b-95a5-cfde99b30719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:58.734237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa3561e-13b5-4e6b-95a5-cfde99b30719 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.734515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa3561e-13b5-4e6b-95a5-cfde99b30719 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.734842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa3561e-13b5-4e6b-95a5-cfde99b30719 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.754266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa3561e-13b5-4e6b-95a5-cfde99b30719 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-9aa3561e-13b5-4e6b-95a5-cfde99b30719 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:58.759629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:58.760614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:58.761452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 286/571] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:41:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:58.762679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa3561e-13b5-4e6b-95a5-cfde99b30719 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:58.796464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:58.808577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:58.829817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eade2be5-38eb-410b-b9a2-062cf3b58a6d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:58.865498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfe2d2f-c71a-4673-aef2-af9ece4f9ebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:58.865734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa3561e-13b5-4e6b-95a5-cfde99b30719 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:58.866112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfe2d2f-c71a-4673-aef2-af9ece4f9ebb None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.866388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfe2d2f-c71a-4673-aef2-af9ece4f9ebb None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.867215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfe2d2f-c71a-4673-aef2-af9ece4f9ebb None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.867847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfe2d2f-c71a-4673-aef2-af9ece4f9ebb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:58.868141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfe2d2f-c71a-4673-aef2-af9ece4f9ebb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:58.869505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfe2d2f-c71a-4673-aef2-af9ece4f9ebb None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:58.880161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 286/572] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:58.880939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfe2d2f-c71a-4673-aef2-af9ece4f9ebb None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:58.903145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4187586-3a6a-41a0-81ad-77bb0b7b1cc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:58.903765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4187586-3a6a-41a0-81ad-77bb0b7b1cc1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.904039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4187586-3a6a-41a0-81ad-77bb0b7b1cc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.904313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4187586-3a6a-41a0-81ad-77bb0b7b1cc1 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.911098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4187586-3a6a-41a0-81ad-77bb0b7b1cc1 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:58.912519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4187586-3a6a-41a0-81ad-77bb0b7b1cc1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f4187586-3a6a-41a0-81ad-77bb0b7b1cc1 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:41:58.924428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:58.925632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 287/573] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:41:58 2022] GET /identity/v3/users/glance => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:58.926904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d78a82-68fb-49b0-88f4-52781d5988bd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:58.927702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d78a82-68fb-49b0-88f4-52781d5988bd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:58.927702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d78a82-68fb-49b0-88f4-52781d5988bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:58.944113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b2d78a82-68fb-49b0-88f4-52781d5988bd None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:41:58.945294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b2d78a82-68fb-49b0-88f4-52781d5988bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:41:58.979826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfe2d2f-c71a-4673-aef2-af9ece4f9ebb None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.009497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 288/574] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:41:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:41:59.010336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d78a82-68fb-49b0-88f4-52781d5988bd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.012538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81faf688-c570-4331-b012-48f586062e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.013910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81faf688-c570-4331-b012-48f586062e75 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.014235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81faf688-c570-4331-b012-48f586062e75 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.014513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81faf688-c570-4331-b012-48f586062e75 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.022490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81faf688-c570-4331-b012-48f586062e75 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.024466 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81faf688-c570-4331-b012-48f586062e75 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:59.034618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:59.037093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-81faf688-c570-4331-b012-48f586062e75 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:41:59.041123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:59.042026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:59.042937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:41:59.042937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:41:59.042937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:41:59.042937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:41:59.042937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:41:59.042937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:41:59.042937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:59.042937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:59.045726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 287/575] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:59.047015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81faf688-c570-4331-b012-48f586062e75 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.121744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d78a82-68fb-49b0-88f4-52781d5988bd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.152343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81faf688-c570-4331-b012-48f586062e75 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.158160 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae82cb5e-a58b-4e26-a544-1a2fc2118a11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.158770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae82cb5e-a58b-4e26-a544-1a2fc2118a11 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.159454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae82cb5e-a58b-4e26-a544-1a2fc2118a11 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.159733 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae82cb5e-a58b-4e26-a544-1a2fc2118a11 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.160389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae82cb5e-a58b-4e26-a544-1a2fc2118a11 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:59.160695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae82cb5e-a58b-4e26-a544-1a2fc2118a11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.162736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae82cb5e-a58b-4e26-a544-1a2fc2118a11 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:59.175467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 289/576] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:59.175467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae82cb5e-a58b-4e26-a544-1a2fc2118a11 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.190379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcb0256-e597-4399-80f7-b46ce99a4b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.191000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcb0256-e597-4399-80f7-b46ce99a4b29 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.191277 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcb0256-e597-4399-80f7-b46ce99a4b29 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.191548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fcb0256-e597-4399-80f7-b46ce99a4b29 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.199992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fcb0256-e597-4399-80f7-b46ce99a4b29 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.202697 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fcb0256-e597-4399-80f7-b46ce99a4b29 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-1fcb0256-e597-4399-80f7-b46ce99a4b29 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:41:59.218496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:59.219553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:59.220647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:41:59.220647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:41:59.220647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:41:59.220647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:41:59.220647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:41:59.220647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:41:59.220647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:41:59.220647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:59.223032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 288/577] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:59.224283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcb0256-e597-4399-80f7-b46ce99a4b29 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.261831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:59.281911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae82cb5e-a58b-4e26-a544-1a2fc2118a11 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.294099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:59.315626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0874f4-f206-44d1-81e0-2715739afaa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.316227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0874f4-f206-44d1-81e0-2715739afaa2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.316499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0874f4-f206-44d1-81e0-2715739afaa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.316765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0874f4-f206-44d1-81e0-2715739afaa2 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.317472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0874f4-f206-44d1-81e0-2715739afaa2 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:59.317769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0874f4-f206-44d1-81e0-2715739afaa2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.319216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0874f4-f206-44d1-81e0-2715739afaa2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:59.334448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fcb0256-e597-4399-80f7-b46ce99a4b29 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.335720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 290/578] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:59.336495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0874f4-f206-44d1-81e0-2715739afaa2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.368991 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67619e23-1d49-4e1d-b4b8-f62d9849db37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.369596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67619e23-1d49-4e1d-b4b8-f62d9849db37 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.369870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67619e23-1d49-4e1d-b4b8-f62d9849db37 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.370176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67619e23-1d49-4e1d-b4b8-f62d9849db37 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.370859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67619e23-1d49-4e1d-b4b8-f62d9849db37 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:59.371162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67619e23-1d49-4e1d-b4b8-f62d9849db37 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.372571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67619e23-1d49-4e1d-b4b8-f62d9849db37 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:59.384777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 289/579] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:59.385649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67619e23-1d49-4e1d-b4b8-f62d9849db37 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.433364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0874f4-f206-44d1-81e0-2715739afaa2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.466483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8e9a0f-afa6-48fa-8670-930945ba9ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.467099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8e9a0f-afa6-48fa-8670-930945ba9ee0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.467375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8e9a0f-afa6-48fa-8670-930945ba9ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.467645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8e9a0f-afa6-48fa-8670-930945ba9ee0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.475680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:59.477928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8e9a0f-afa6-48fa-8670-930945ba9ee0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.479735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8e9a0f-afa6-48fa-8670-930945ba9ee0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:59.480127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67619e23-1d49-4e1d-b4b8-f62d9849db37 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-aa8e9a0f-afa6-48fa-8670-930945ba9ee0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:59.494449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:59.495566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:59.496692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:59.496692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:59.496692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:59.496692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 291/580] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:59.496692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8e9a0f-afa6-48fa-8670-930945ba9ee0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.503074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:59.518154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f383febd-fa5c-4aa9-913d-3aad8ad99443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.518952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f383febd-fa5c-4aa9-913d-3aad8ad99443 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.519221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f383febd-fa5c-4aa9-913d-3aad8ad99443 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.519485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f383febd-fa5c-4aa9-913d-3aad8ad99443 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.520161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f383febd-fa5c-4aa9-913d-3aad8ad99443 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:59.520459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f383febd-fa5c-4aa9-913d-3aad8ad99443 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.521868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f383febd-fa5c-4aa9-913d-3aad8ad99443 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:59.533325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 290/581] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:59.534342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f383febd-fa5c-4aa9-913d-3aad8ad99443 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.597494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8e9a0f-afa6-48fa-8670-930945ba9ee0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.630279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f383febd-fa5c-4aa9-913d-3aad8ad99443 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.630799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7b2e0d-71d1-4636-bb66-ed13983ff592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.631414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7b2e0d-71d1-4636-bb66-ed13983ff592 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.631692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7b2e0d-71d1-4636-bb66-ed13983ff592 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.631961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7b2e0d-71d1-4636-bb66-ed13983ff592 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.642542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7b2e0d-71d1-4636-bb66-ed13983ff592 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fe7b2e0d-71d1-4636-bb66-ed13983ff592 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:41:59.647705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:59.649147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 292/582] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:59.649997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7b2e0d-71d1-4636-bb66-ed13983ff592 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.668158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f892434b-ed5d-49b1-9588-de470874f2e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.668781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f892434b-ed5d-49b1-9588-de470874f2e1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.669059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f892434b-ed5d-49b1-9588-de470874f2e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.669333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f892434b-ed5d-49b1-9588-de470874f2e1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.680139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f892434b-ed5d-49b1-9588-de470874f2e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.681550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f892434b-ed5d-49b1-9588-de470874f2e1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-f892434b-ed5d-49b1-9588-de470874f2e1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:41:59.695014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:41:59.696042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:41:59.696963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:41:59.696963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:41:59.696963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:41:59.696963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 291/583] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:41:59.696963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f892434b-ed5d-49b1-9588-de470874f2e1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.704404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:59.727102 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:59.750974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7b2e0d-71d1-4636-bb66-ed13983ff592 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.783902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd6d486-0898-4692-b5e4-669107b66f44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6d486-0898-4692-b5e4-669107b66f44 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.784751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6d486-0898-4692-b5e4-669107b66f44 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.785018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6d486-0898-4692-b5e4-669107b66f44 None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/dad93d99b865407aa120035015f4692a/roles/2c1bcaa088914d4593bb1debe2fc88d6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.795038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f892434b-ed5d-49b1-9588-de470874f2e1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.799493 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd6d486-0898-4692-b5e4-669107b66f44 None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=dad93d99b865407aa120035015f4692a, role_id=2c1bcaa088914d4593bb1debe2fc88d6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.800926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd6d486-0898-4692-b5e4-669107b66f44 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:59.815532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 293/584] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:41:59 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/dad93d99b865407aa120035015f4692a/roles/2c1bcaa088914d4593bb1debe2fc88d6 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:41:59.816325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd6d486-0898-4692-b5e4-669107b66f44 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.830418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273b2d67-b155-4607-beb5-4bcd22ae4e1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.831273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b2d67-b155-4607-beb5-4bcd22ae4e1c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.831471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b2d67-b155-4607-beb5-4bcd22ae4e1c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.831789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b2d67-b155-4607-beb5-4bcd22ae4e1c None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.832642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273b2d67-b155-4607-beb5-4bcd22ae4e1c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:59.832949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273b2d67-b155-4607-beb5-4bcd22ae4e1c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.834376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273b2d67-b155-4607-beb5-4bcd22ae4e1c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:41:59.848454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 292/585] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:41:59.849195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273b2d67-b155-4607-beb5-4bcd22ae4e1c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:41:59.947402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd6d486-0898-4692-b5e4-669107b66f44 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.949150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:59.953351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:41:59.963998 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273b2d67-b155-4607-beb5-4bcd22ae4e1c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:41:59.988628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8284f0-d517-4879-a2e0-f08186794489 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:41:59.989240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8284f0-d517-4879-a2e0-f08186794489 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:41:59.989519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8284f0-d517-4879-a2e0-f08186794489 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:41:59.989791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8284f0-d517-4879-a2e0-f08186794489 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:41:59.990555 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8284f0-d517-4879-a2e0-f08186794489 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=db6c9dbbcf244a6f8bac59d2c9c973bc) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:41:59.990864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8284f0-d517-4879-a2e0-f08186794489 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:41:59.992385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8284f0-d517-4879-a2e0-f08186794489 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:00.003622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 294/586] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/role_assignments?user.id=db6c9dbbcf244a6f8bac59d2c9c973bc&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 259 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:00.004497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8284f0-d517-4879-a2e0-f08186794489 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:00.007160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1510a655-f657-4777-9e62-cdb1215143e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:00.008714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1510a655-f657-4777-9e62-cdb1215143e6 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:00.008991 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1510a655-f657-4777-9e62-cdb1215143e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:00.010185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1510a655-f657-4777-9e62-cdb1215143e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:00.010875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1510a655-f657-4777-9e62-cdb1215143e6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:00.011176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1510a655-f657-4777-9e62-cdb1215143e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:00.012574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1510a655-f657-4777-9e62-cdb1215143e6 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:00.027496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 293/587] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:41:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:00.028188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1510a655-f657-4777-9e62-cdb1215143e6 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:00.161405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8284f0-d517-4879-a2e0-f08186794489 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:00.161655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1510a655-f657-4777-9e62-cdb1215143e6 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:00.200092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31da5694-f12f-4ee4-befc-60ff1ea1a57b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:00.200938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31da5694-f12f-4ee4-befc-60ff1ea1a57b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:00.201352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31da5694-f12f-4ee4-befc-60ff1ea1a57b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:00.201759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31da5694-f12f-4ee4-befc-60ff1ea1a57b None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:00.202606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31da5694-f12f-4ee4-befc-60ff1ea1a57b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:00.203045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31da5694-f12f-4ee4-befc-60ff1ea1a57b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:00.203409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e243c1-7caf-433d-899e-b8fd4e0d720f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:00.204096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e243c1-7caf-433d-899e-b8fd4e0d720f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:00.204441 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e243c1-7caf-433d-899e-b8fd4e0d720f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:00.204818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e243c1-7caf-433d-899e-b8fd4e0d720f None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:00.205212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31da5694-f12f-4ee4-befc-60ff1ea1a57b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:00.212832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e243c1-7caf-433d-899e-b8fd4e0d720f None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:00.214343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e243c1-7caf-433d-899e-b8fd4e0d720f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:00.219246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 295/588] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:00.220258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31da5694-f12f-4ee4-befc-60ff1ea1a57b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-24e243c1-7caf-433d-899e-b8fd4e0d720f None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:00.227409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:00.228562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 294/589] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:42:00 2022] GET /identity/v3/users/cinder => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:00.229646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e243c1-7caf-433d-899e-b8fd4e0d720f None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:00.251950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:00.266550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:00.333926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31da5694-f12f-4ee4-befc-60ff1ea1a57b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:00.341251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e243c1-7caf-433d-899e-b8fd4e0d720f None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:00.370411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d56904d-f4ee-470f-8e88-d959857a8ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:00.371279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d56904d-f4ee-470f-8e88-d959857a8ed8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:00.371760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d56904d-f4ee-470f-8e88-d959857a8ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:00.372200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d56904d-f4ee-470f-8e88-d959857a8ed8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:00.377860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9f4ed8-352a-41ce-9096-c0c02b6e86e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:00.378516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9f4ed8-352a-41ce-9096-c0c02b6e86e5 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:00.378789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9f4ed8-352a-41ce-9096-c0c02b6e86e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:00.379056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9f4ed8-352a-41ce-9096-c0c02b6e86e5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:00.384138 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d56904d-f4ee-470f-8e88-d959857a8ed8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:00.385762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d56904d-f4ee-470f-8e88-d959857a8ed8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:00.386164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9f4ed8-352a-41ce-9096-c0c02b6e86e5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:00.387706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9f4ed8-352a-41ce-9096-c0c02b6e86e5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-1b9f4ed8-352a-41ce-9096-c0c02b6e86e5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:00.401198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:00.402140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:00.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:00.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:00.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:00.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:00.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:00.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:00.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:00.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:00.403025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 295/590] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:00.403682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9f4ed8-352a-41ce-9096-c0c02b6e86e5 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0d56904d-f4ee-470f-8e88-d959857a8ed8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:00.404684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:00.405572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:00.406478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:00.406478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:00.406478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:00.408308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 296/591] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:00.409315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d56904d-f4ee-470f-8e88-d959857a8ed8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:00.489334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:00.498355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:00.524865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9f4ed8-352a-41ce-9096-c0c02b6e86e5 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:00.526192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d56904d-f4ee-470f-8e88-d959857a8ed8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:00.566403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e55beb-f136-44ad-9189-3c33183a2018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:00.566737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad778cf-0332-4fe5-80a4-e9d4aa166e64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:00.567015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e55beb-f136-44ad-9189-3c33183a2018 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:00.567247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad778cf-0332-4fe5-80a4-e9d4aa166e64 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:00.567347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e55beb-f136-44ad-9189-3c33183a2018 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:00.567522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad778cf-0332-4fe5-80a4-e9d4aa166e64 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:00.567612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e55beb-f136-44ad-9189-3c33183a2018 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:00.567797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad778cf-0332-4fe5-80a4-e9d4aa166e64 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:00.568229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e55beb-f136-44ad-9189-3c33183a2018 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:00.568517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad778cf-0332-4fe5-80a4-e9d4aa166e64 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:00.568626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e55beb-f136-44ad-9189-3c33183a2018 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:00.568819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad778cf-0332-4fe5-80a4-e9d4aa166e64 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:00.570083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e55beb-f136-44ad-9189-3c33183a2018 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:00.570331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad778cf-0332-4fe5-80a4-e9d4aa166e64 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:00.583710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 297/592] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:00.584807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e55beb-f136-44ad-9189-3c33183a2018 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:00.593591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 296/593] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Oct 2 00:42:00 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:00.594481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad778cf-0332-4fe5-80a4-e9d4aa166e64 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:00.694489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e55beb-f136-44ad-9189-3c33183a2018 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:00.709340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad778cf-0332-4fe5-80a4-e9d4aa166e64 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:00.732942 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b34aee-1225-4473-908d-a688ea6c7379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:00.733598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b34aee-1225-4473-908d-a688ea6c7379 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:00.734006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b34aee-1225-4473-908d-a688ea6c7379 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:00.734435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b34aee-1225-4473-908d-a688ea6c7379 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:00.735091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b34aee-1225-4473-908d-a688ea6c7379 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:00.735393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b34aee-1225-4473-908d-a688ea6c7379 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:00.737589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b34aee-1225-4473-908d-a688ea6c7379 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:00.744734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:00.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:00.749597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406d1f6c-f41c-4a10-9c25-0aae3c09ed4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:00.750567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406d1f6c-f41c-4a10-9c25-0aae3c09ed4a None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:00.751686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406d1f6c-f41c-4a10-9c25-0aae3c09ed4a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:00.752160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406d1f6c-f41c-4a10-9c25-0aae3c09ed4a None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/30fc59f80f544cef8b3869e8d37bb341/roles/2c1bcaa088914d4593bb1debe2fc88d6` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:00.758131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 298/594] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:00.760307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b34aee-1225-4473-908d-a688ea6c7379 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:00.772056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406d1f6c-f41c-4a10-9c25-0aae3c09ed4a None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=30fc59f80f544cef8b3869e8d37bb341, role_id=2c1bcaa088914d4593bb1debe2fc88d6)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:00.773730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406d1f6c-f41c-4a10-9c25-0aae3c09ed4a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:00.790213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 297/595] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:42:00 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/30fc59f80f544cef8b3869e8d37bb341/roles/2c1bcaa088914d4593bb1debe2fc88d6 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:42:00.798462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406d1f6c-f41c-4a10-9c25-0aae3c09ed4a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:00.895849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b34aee-1225-4473-908d-a688ea6c7379 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:00.979116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf2a917-1a97-4b7b-894c-7795de0b471f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:00.979763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf2a917-1a97-4b7b-894c-7795de0b471f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:00.980037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf2a917-1a97-4b7b-894c-7795de0b471f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:00.980300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf2a917-1a97-4b7b-894c-7795de0b471f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:01.002188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf2a917-1a97-4b7b-894c-7795de0b471f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:01.003672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf2a917-1a97-4b7b-894c-7795de0b471f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:01.016435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406d1f6c-f41c-4a10-9c25-0aae3c09ed4a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0cf2a917-1a97-4b7b-894c-7795de0b471f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:01.032564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:01.033561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:01.034473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:01.034473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:01.034473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:01.040570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 299/596] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:01.040895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf2a917-1a97-4b7b-894c-7795de0b471f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:01.078772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948bd6dc-d2f7-4a22-8738-149c84c735dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:01.079404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948bd6dc-d2f7-4a22-8738-149c84c735dc None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:01.079783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948bd6dc-d2f7-4a22-8738-149c84c735dc None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:01.080052 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948bd6dc-d2f7-4a22-8738-149c84c735dc None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:01.088239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948bd6dc-d2f7-4a22-8738-149c84c735dc None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:01.089784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948bd6dc-d2f7-4a22-8738-149c84c735dc None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:01.100779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-948bd6dc-d2f7-4a22-8738-149c84c735dc None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:01.104793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:01.105726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:01.106712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:01.106712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:01.106712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:01.106712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:01.106712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:01.106712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:01.106712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:01.106712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:01.106712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 02 00:42:01.106712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:01.107237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 298/597] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:00 2022] GET /identity/v3/users/neutron => generated 85 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:01.113631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948bd6dc-d2f7-4a22-8738-149c84c735dc None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:01.200171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:01.221584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf2a917-1a97-4b7b-894c-7795de0b471f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:01.291816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d30ec10-3501-41ba-94b1-356cec97da8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:01.292424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d30ec10-3501-41ba-94b1-356cec97da8c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:01.292700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d30ec10-3501-41ba-94b1-356cec97da8c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:01.292972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d30ec10-3501-41ba-94b1-356cec97da8c None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:01.293610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d30ec10-3501-41ba-94b1-356cec97da8c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:01.293911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d30ec10-3501-41ba-94b1-356cec97da8c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:01.295932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d30ec10-3501-41ba-94b1-356cec97da8c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:01.302794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948bd6dc-d2f7-4a22-8738-149c84c735dc None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:01.318293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 300/598] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:01.324953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d30ec10-3501-41ba-94b1-356cec97da8c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:01.372506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263f6380-3dcc-4603-bd62-c6fb19527bd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:01.373117 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263f6380-3dcc-4603-bd62-c6fb19527bd0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:01.373395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263f6380-3dcc-4603-bd62-c6fb19527bd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:01.373669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263f6380-3dcc-4603-bd62-c6fb19527bd0 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:01.374426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263f6380-3dcc-4603-bd62-c6fb19527bd0 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:01.374730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263f6380-3dcc-4603-bd62-c6fb19527bd0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:01.376155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263f6380-3dcc-4603-bd62-c6fb19527bd0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:01.415080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 299/599] 10.222.0.25 () {56 vars in 1139 bytes} [Sun Oct 2 00:42:01 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:01.419029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263f6380-3dcc-4603-bd62-c6fb19527bd0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:01.503505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d30ec10-3501-41ba-94b1-356cec97da8c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:01.518131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:01.546965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8587b2c2-8de4-401a-96b8-d6dd17d59f98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:01.548261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8587b2c2-8de4-401a-96b8-d6dd17d59f98 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:01.548688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8587b2c2-8de4-401a-96b8-d6dd17d59f98 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:01.549131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8587b2c2-8de4-401a-96b8-d6dd17d59f98 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:01.557036 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8587b2c2-8de4-401a-96b8-d6dd17d59f98 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:01.558844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8587b2c2-8de4-401a-96b8-d6dd17d59f98 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-8587b2c2-8de4-401a-96b8-d6dd17d59f98 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:01.572315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:01.573421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:01.574563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:01.574563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:01.574563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:01.574563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:01.574563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:01.574563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:01.574563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:01.574563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:01.576945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 301/600] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:01.582700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8587b2c2-8de4-401a-96b8-d6dd17d59f98 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:01.628212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:01.634523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263f6380-3dcc-4603-bd62-c6fb19527bd0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:01.702835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c3b5d5-eeec-4b35-a255-bb65a5f4f393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:01.703467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c3b5d5-eeec-4b35-a255-bb65a5f4f393 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:01.703737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c3b5d5-eeec-4b35-a255-bb65a5f4f393 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:01.704004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c3b5d5-eeec-4b35-a255-bb65a5f4f393 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:01.716484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c3b5d5-eeec-4b35-a255-bb65a5f4f393 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:01.717941 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c3b5d5-eeec-4b35-a255-bb65a5f4f393 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:01.737717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8587b2c2-8de4-401a-96b8-d6dd17d59f98 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-d6c3b5d5-eeec-4b35-a255-bb65a5f4f393 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:01.749847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:01.751160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:01.752034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:01.752034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:01.752034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:01.755807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 300/601] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:01.757696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c3b5d5-eeec-4b35-a255-bb65a5f4f393 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:01.791680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d1e148-4e90-478b-8298-2decde6279d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:01.792645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d1e148-4e90-478b-8298-2decde6279d5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:01.793120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d1e148-4e90-478b-8298-2decde6279d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:01.793609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d1e148-4e90-478b-8298-2decde6279d5 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:01.794533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d1e148-4e90-478b-8298-2decde6279d5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:01.795034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d1e148-4e90-478b-8298-2decde6279d5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:01.796662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d1e148-4e90-478b-8298-2decde6279d5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:01.809899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 302/602] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:01.815926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d1e148-4e90-478b-8298-2decde6279d5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:01.850316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:01.906245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c3b5d5-eeec-4b35-a255-bb65a5f4f393 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:01.947782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d1e148-4e90-478b-8298-2decde6279d5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:01.953640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59703734-66f8-4fc2-a460-e57ec3790657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:01.954852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59703734-66f8-4fc2-a460-e57ec3790657 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:01.955139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59703734-66f8-4fc2-a460-e57ec3790657 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:01.955421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59703734-66f8-4fc2-a460-e57ec3790657 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:01.956096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59703734-66f8-4fc2-a460-e57ec3790657 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:01.956405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59703734-66f8-4fc2-a460-e57ec3790657 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:01.957970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59703734-66f8-4fc2-a460-e57ec3790657 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:01.984123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682dfda9-3b9b-4574-bf28-e32d9d89afef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:01.984505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 301/603] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:01.985389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682dfda9-3b9b-4574-bf28-e32d9d89afef None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:01.985887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682dfda9-3b9b-4574-bf28-e32d9d89afef None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:01.986392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682dfda9-3b9b-4574-bf28-e32d9d89afef None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/eb325d3c277b45c2a5c4e0e94078be29/roles/2c1bcaa088914d4593bb1debe2fc88d6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:01.990039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59703734-66f8-4fc2-a460-e57ec3790657 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:02.003220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682dfda9-3b9b-4574-bf28-e32d9d89afef None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=eb325d3c277b45c2a5c4e0e94078be29, role_id=2c1bcaa088914d4593bb1debe2fc88d6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:02.005048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682dfda9-3b9b-4574-bf28-e32d9d89afef None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:02.021950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:02.033352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 303/604] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:42:01 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/eb325d3c277b45c2a5c4e0e94078be29/roles/2c1bcaa088914d4593bb1debe2fc88d6 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:42:02.064264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861ca720-4b58-4768-87e7-24089050d919 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:02.064544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861ca720-4b58-4768-87e7-24089050d919 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:02.064820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861ca720-4b58-4768-87e7-24089050d919 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:02.074298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 304/605] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:02 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:02.079939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd65022-487b-401d-8650-9208179b232b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:02.080214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd65022-487b-401d-8650-9208179b232b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:02.080488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd65022-487b-401d-8650-9208179b232b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:02.117075 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8bd65022-487b-401d-8650-9208179b232b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:02.118342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8bd65022-487b-401d-8650-9208179b232b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:02.221255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59703734-66f8-4fc2-a460-e57ec3790657 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:02.259745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 305/606] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:42:02.266325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d30011f-51d9-4bfc-b98c-d470a7d347d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:02.266958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d30011f-51d9-4bfc-b98c-d470a7d347d7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:02.267238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d30011f-51d9-4bfc-b98c-d470a7d347d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:02.267511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d30011f-51d9-4bfc-b98c-d470a7d347d7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:02.275233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d30011f-51d9-4bfc-b98c-d470a7d347d7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:02.276687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d30011f-51d9-4bfc-b98c-d470a7d347d7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-0d30011f-51d9-4bfc-b98c-d470a7d347d7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:02.288287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:02.289739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:02.290711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:02.290711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:02.290711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:02.290711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:02.290711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:02.290711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:02.290711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:02.290711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:02.294182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 302/607] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:02.295889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd65022-487b-401d-8650-9208179b232b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:02.308314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:02.404148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd65022-487b-401d-8650-9208179b232b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:02.456138 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723ecfd6-c2ef-4ed2-9de7-3e25afc52c64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:02.456769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723ecfd6-c2ef-4ed2-9de7-3e25afc52c64 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:02.457097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723ecfd6-c2ef-4ed2-9de7-3e25afc52c64 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:02.457373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723ecfd6-c2ef-4ed2-9de7-3e25afc52c64 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:02.458089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723ecfd6-c2ef-4ed2-9de7-3e25afc52c64 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:02.458403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723ecfd6-c2ef-4ed2-9de7-3e25afc52c64 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:02.459874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723ecfd6-c2ef-4ed2-9de7-3e25afc52c64 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:02.478118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 306/608] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:02.484196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d30011f-51d9-4bfc-b98c-d470a7d347d7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:02.532785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723ecfd6-c2ef-4ed2-9de7-3e25afc52c64 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:02.592162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:02.602901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d30011f-51d9-4bfc-b98c-d470a7d347d7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:02.645300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:02.649971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536d1d27-8cff-492d-b076-fd33003744f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:02.651166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d1d27-8cff-492d-b076-fd33003744f0 None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:02.651793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d1d27-8cff-492d-b076-fd33003744f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:02.652221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536d1d27-8cff-492d-b076-fd33003744f0 None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/e75811ee9ed7490985fed336ab9feb19/roles/2c1bcaa088914d4593bb1debe2fc88d6` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:02.675144 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536d1d27-8cff-492d-b076-fd33003744f0 None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=e75811ee9ed7490985fed336ab9feb19, role_id=2c1bcaa088914d4593bb1debe2fc88d6)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:02.676695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536d1d27-8cff-492d-b076-fd33003744f0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:02.685575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723ecfd6-c2ef-4ed2-9de7-3e25afc52c64 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:02.692238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 303/609] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:42:02 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/e75811ee9ed7490985fed336ab9feb19/roles/2c1bcaa088914d4593bb1debe2fc88d6 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:42:02.693590 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e038742-bc0e-4784-a9d7-4afc5c31e398 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:02.693922 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e038742-bc0e-4784-a9d7-4afc5c31e398 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:02.694257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e038742-bc0e-4784-a9d7-4afc5c31e398 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:02.695852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 304/610] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:02.701987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfce4b3-3472-46fc-98f6-397887986c3c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:02.702628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfce4b3-3472-46fc-98f6-397887986c3c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:02.703105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfce4b3-3472-46fc-98f6-397887986c3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:02.724279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9dfce4b3-3472-46fc-98f6-397887986c3c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:02.726005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9dfce4b3-3472-46fc-98f6-397887986c3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:02.778119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b89f719-e349-42b0-8096-e5c41b7bb2ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:02.778816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b89f719-e349-42b0-8096-e5c41b7bb2ae None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:02.779098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b89f719-e349-42b0-8096-e5c41b7bb2ae None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:02.779369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b89f719-e349-42b0-8096-e5c41b7bb2ae None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:02.803468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 305/611] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 02 00:42:02.815182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b89f719-e349-42b0-8096-e5c41b7bb2ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6b89f719-e349-42b0-8096-e5c41b7bb2ae None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:02.820683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:02.821636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:02.822514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:02.827525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 307/612] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:02.833063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfce4b3-3472-46fc-98f6-397887986c3c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:02.924371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:02.968778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfce4b3-3472-46fc-98f6-397887986c3c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:03.003504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f102035-24dd-4e50-a6d6-20413e1685bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:03.005365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f102035-24dd-4e50-a6d6-20413e1685bf None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:03.005365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f102035-24dd-4e50-a6d6-20413e1685bf None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:03.005365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f102035-24dd-4e50-a6d6-20413e1685bf None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:03.005365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f102035-24dd-4e50-a6d6-20413e1685bf None admin] RBAC: Adding query filter params (name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:03.005789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f102035-24dd-4e50-a6d6-20413e1685bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:03.007888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f102035-24dd-4e50-a6d6-20413e1685bf None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:03.016877 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 306/613] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:42:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:03.022493 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b89f719-e349-42b0-8096-e5c41b7bb2ae None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:03.183977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b89f719-e349-42b0-8096-e5c41b7bb2ae None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:03.224443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:03.231699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d90e174-a7d9-4cd7-9c80-d2a719c23b10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:03.232339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d90e174-a7d9-4cd7-9c80-d2a719c23b10 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:03.232617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d90e174-a7d9-4cd7-9c80-d2a719c23b10 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:03.232889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d90e174-a7d9-4cd7-9c80-d2a719c23b10 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:03.247592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d90e174-a7d9-4cd7-9c80-d2a719c23b10 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:03.249054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d90e174-a7d9-4cd7-9c80-d2a719c23b10 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1d90e174-a7d9-4cd7-9c80-d2a719c23b10 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:03.267748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:03.268965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:03.269869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:03.269869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:03.269869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:03.273147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 308/614] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:03.274605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f102035-24dd-4e50-a6d6-20413e1685bf None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:03.319532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d90e174-a7d9-4cd7-9c80-d2a719c23b10 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:03.383896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f102035-24dd-4e50-a6d6-20413e1685bf None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:03.406427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:03.435475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a9cc4-6572-4c71-88b9-5584b4ae7836 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:03.436008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a9cc4-6572-4c71-88b9-5584b4ae7836 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:03.436286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a9cc4-6572-4c71-88b9-5584b4ae7836 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:03.436558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a9cc4-6572-4c71-88b9-5584b4ae7836 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:03.437202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9a9cc4-6572-4c71-88b9-5584b4ae7836 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:03.437504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9a9cc4-6572-4c71-88b9-5584b4ae7836 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:03.438894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9a9cc4-6572-4c71-88b9-5584b4ae7836 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:03.466169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 307/615] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:03.467114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a9cc4-6572-4c71-88b9-5584b4ae7836 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:03.532372 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d90e174-a7d9-4cd7-9c80-d2a719c23b10 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:03.589024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46f9319-ccc3-456e-8470-1fdd36115daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:03.589664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46f9319-ccc3-456e-8470-1fdd36115daa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:03.589941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46f9319-ccc3-456e-8470-1fdd36115daa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:03.590242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46f9319-ccc3-456e-8470-1fdd36115daa None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:03.592851 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a9cc4-6572-4c71-88b9-5584b4ae7836 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:03.612106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46f9319-ccc3-456e-8470-1fdd36115daa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a46f9319-ccc3-456e-8470-1fdd36115daa None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:03.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:03.620722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:03.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:03.627263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 309/616] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:03.627263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46f9319-ccc3-456e-8470-1fdd36115daa None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:03.659076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa44163-444a-4bcc-a3b2-87a9e9e071c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:03.659714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa44163-444a-4bcc-a3b2-87a9e9e071c2 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:03.659992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa44163-444a-4bcc-a3b2-87a9e9e071c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:03.660265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa44163-444a-4bcc-a3b2-87a9e9e071c2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:03.669990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa44163-444a-4bcc-a3b2-87a9e9e071c2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:03.671796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa44163-444a-4bcc-a3b2-87a9e9e071c2 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-8fa44163-444a-4bcc-a3b2-87a9e9e071c2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:03.685132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:03.686213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:03.687181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:03.687181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:03.687181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:03.687181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:03.687181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:03.687181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:03.687181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:03.687181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:03.687181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:42:03.687181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:03.687720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:03.689928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 308/617] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 2 00:42:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:03.691406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99c871c-733a-4da2-a9a6-0fada02052f2 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:03.691788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99c871c-733a-4da2-a9a6-0fada02052f2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:03.692157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99c871c-733a-4da2-a9a6-0fada02052f2 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:03.697720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 309/618] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:03.698636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99c871c-733a-4da2-a9a6-0fada02052f2 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:03.735595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:03.770492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46f9319-ccc3-456e-8470-1fdd36115daa None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:03.809729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99c871c-733a-4da2-a9a6-0fada02052f2 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:03.814911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439840e8-0745-4065-961b-15048cb500df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:03.815509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439840e8-0745-4065-961b-15048cb500df None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:03.815781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439840e8-0745-4065-961b-15048cb500df None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:03.816044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439840e8-0745-4065-961b-15048cb500df None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:03.816747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439840e8-0745-4065-961b-15048cb500df None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:03.817051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439840e8-0745-4065-961b-15048cb500df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:03.818959 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439840e8-0745-4065-961b-15048cb500df None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:03.830569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 310/619] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:42:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:03.831868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b16535-ddf3-49a7-9c11-c4f927104f3a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:03.832695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b16535-ddf3-49a7-9c11-c4f927104f3a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:03.832970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b16535-ddf3-49a7-9c11-c4f927104f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:03.848516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f15c824-7af1-47d4-ac31-7621332fa981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:03.849333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f15c824-7af1-47d4-ac31-7621332fa981 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:03.849755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f15c824-7af1-47d4-ac31-7621332fa981 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:03.850286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f15c824-7af1-47d4-ac31-7621332fa981 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:03.851799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f15c824-7af1-47d4-ac31-7621332fa981 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:03.852230 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f15c824-7af1-47d4-ac31-7621332fa981 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:03.853872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f15c824-7af1-47d4-ac31-7621332fa981 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:03.856520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-22b16535-ddf3-49a7-9c11-c4f927104f3a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:03.858824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-22b16535-ddf3-49a7-9c11-c4f927104f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:03.880804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 310/620] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 2 00:42:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:03.881823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f15c824-7af1-47d4-ac31-7621332fa981 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:03.934325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 311/621] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:03.935356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b16535-ddf3-49a7-9c11-c4f927104f3a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:03.988412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:03.997790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:03.998926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f15c824-7af1-47d4-ac31-7621332fa981 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:04.042145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35059f54-a67b-4ec2-98f2-d429004af10f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:04.042145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35059f54-a67b-4ec2-98f2-d429004af10f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.042145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35059f54-a67b-4ec2-98f2-d429004af10f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.042145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35059f54-a67b-4ec2-98f2-d429004af10f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.062270 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35059f54-a67b-4ec2-98f2-d429004af10f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:04.063709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35059f54-a67b-4ec2-98f2-d429004af10f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-35059f54-a67b-4ec2-98f2-d429004af10f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:04.093393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:04.094438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:04.095317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:04.095317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:04.095317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:04.102934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 311/622] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:04.103769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35059f54-a67b-4ec2-98f2-d429004af10f None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:04.120420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b16535-ddf3-49a7-9c11-c4f927104f3a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:04.158871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd45144a-fe2a-46b9-bfa6-d01a6162c6ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:04.159713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45144a-fe2a-46b9-bfa6-d01a6162c6ff None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.159989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45144a-fe2a-46b9-bfa6-d01a6162c6ff None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.160260 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45144a-fe2a-46b9-bfa6-d01a6162c6ff None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.238087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd45144a-fe2a-46b9-bfa6-d01a6162c6ff None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:04.239617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd45144a-fe2a-46b9-bfa6-d01a6162c6ff None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:04.282535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:04.292190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35059f54-a67b-4ec2-98f2-d429004af10f None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:04.329806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040b6f95-2b5f-4881-ba80-893e7bdfb9e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:04.330459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040b6f95-2b5f-4881-ba80-893e7bdfb9e5 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.330739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040b6f95-2b5f-4881-ba80-893e7bdfb9e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.331013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040b6f95-2b5f-4881-ba80-893e7bdfb9e5 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.331652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040b6f95-2b5f-4881-ba80-893e7bdfb9e5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:04.331952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040b6f95-2b5f-4881-ba80-893e7bdfb9e5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:04.333307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040b6f95-2b5f-4881-ba80-893e7bdfb9e5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:04.346650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 312/623] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:04.347494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040b6f95-2b5f-4881-ba80-893e7bdfb9e5 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:04.351941 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:04.359152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 312/624] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:42:04.360547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd45144a-fe2a-46b9-bfa6-d01a6162c6ff None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:04.461533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd45144a-fe2a-46b9-bfa6-d01a6162c6ff None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:04.482511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040b6f95-2b5f-4881-ba80-893e7bdfb9e5 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:04.505250 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9560d9bc-3e4b-4eff-9d6f-873f0d6ca422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:04.506504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9560d9bc-3e4b-4eff-9d6f-873f0d6ca422 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.507741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9560d9bc-3e4b-4eff-9d6f-873f0d6ca422 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.508499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9560d9bc-3e4b-4eff-9d6f-873f0d6ca422 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.511351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:04.519979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9560d9bc-3e4b-4eff-9d6f-873f0d6ca422 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:04.520332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef68ed3-d92a-4291-b986-f379fc9801dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:04.520957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef68ed3-d92a-4291-b986-f379fc9801dd None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.521228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef68ed3-d92a-4291-b986-f379fc9801dd None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.521496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef68ed3-d92a-4291-b986-f379fc9801dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.522661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9560d9bc-3e4b-4eff-9d6f-873f0d6ca422 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:04.532129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef68ed3-d92a-4291-b986-f379fc9801dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-9560d9bc-3e4b-4eff-9d6f-873f0d6ca422 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:04.536187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:04.537105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:04.538046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:04.538046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:04.538046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:04.538046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:04.538046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:04.538046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:04.538046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:04.538046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:04.538046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:42:04.538046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-4ef68ed3-d92a-4291-b986-f379fc9801dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:04.539157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:04.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 313/625] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:04.540856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef68ed3-d92a-4291-b986-f379fc9801dd None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:04.546004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 313/626] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:04 2022] GET /identity/v3/users/swift => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:04.547205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9560d9bc-3e4b-4eff-9d6f-873f0d6ca422 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:04.622815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:04.659007 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9560d9bc-3e4b-4eff-9d6f-873f0d6ca422 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:04.675753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef68ed3-d92a-4291-b986-f379fc9801dd None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:04.695674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b754c11d-23b3-4f24-a4b1-15b370ddc0b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:04.696272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b754c11d-23b3-4f24-a4b1-15b370ddc0b3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.696602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b754c11d-23b3-4f24-a4b1-15b370ddc0b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.696870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b754c11d-23b3-4f24-a4b1-15b370ddc0b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.697572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b754c11d-23b3-4f24-a4b1-15b370ddc0b3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:04.697874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b754c11d-23b3-4f24-a4b1-15b370ddc0b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:04.699262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b754c11d-23b3-4f24-a4b1-15b370ddc0b3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:04.707341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 314/627] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:42:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:04.708320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b754c11d-23b3-4f24-a4b1-15b370ddc0b3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:04.718488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc63297d-1e75-417d-90ef-2aee1d067ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:04.719121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc63297d-1e75-417d-90ef-2aee1d067ba0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.719752 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc63297d-1e75-417d-90ef-2aee1d067ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.720029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc63297d-1e75-417d-90ef-2aee1d067ba0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.730757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc63297d-1e75-417d-90ef-2aee1d067ba0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:04.732184 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc63297d-1e75-417d-90ef-2aee1d067ba0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-dc63297d-1e75-417d-90ef-2aee1d067ba0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:04.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:04.748735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:04.754912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:04.754912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:04.754912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:04.754912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 314/628] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:04.754912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302850d9-7985-4da7-9832-537e7c66af62 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.754912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302850d9-7985-4da7-9832-537e7c66af62 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.754912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302850d9-7985-4da7-9832-537e7c66af62 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.754912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 315/629] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:04 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:04.754912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302850d9-7985-4da7-9832-537e7c66af62 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:04.774979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:04.820403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b754c11d-23b3-4f24-a4b1-15b370ddc0b3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:04.853797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbd40c6-7aa3-4445-9995-b135e094f2e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:04.854432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbd40c6-7aa3-4445-9995-b135e094f2e4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.854708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbd40c6-7aa3-4445-9995-b135e094f2e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.854978 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbd40c6-7aa3-4445-9995-b135e094f2e4 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.855703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbd40c6-7aa3-4445-9995-b135e094f2e4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:04.856419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbd40c6-7aa3-4445-9995-b135e094f2e4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:04.857852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbd40c6-7aa3-4445-9995-b135e094f2e4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:04.869061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302850d9-7985-4da7-9832-537e7c66af62 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:04.880665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 315/630] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 2 00:42:04 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:04.881546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbd40c6-7aa3-4445-9995-b135e094f2e4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:04.887854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:04.910434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9935cf-5d5f-495c-a811-27bd1232c3f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:04.911538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9935cf-5d5f-495c-a811-27bd1232c3f5 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.911967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9935cf-5d5f-495c-a811-27bd1232c3f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.912395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9935cf-5d5f-495c-a811-27bd1232c3f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.931628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9935cf-5d5f-495c-a811-27bd1232c3f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:04.931628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9935cf-5d5f-495c-a811-27bd1232c3f5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-9e9935cf-5d5f-495c-a811-27bd1232c3f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:04.943138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:04.944209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:04.945110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:04.945110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:04.945110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:04.945110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 316/631] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:04.945654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263ed451-e392-47ba-abdb-6507c382f2c7 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:04.945811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263ed451-e392-47ba-abdb-6507c382f2c7 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:04.946068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263ed451-e392-47ba-abdb-6507c382f2c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:04.964677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-263ed451-e392-47ba-abdb-6507c382f2c7 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:04.965838 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-263ed451-e392-47ba-abdb-6507c382f2c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:04.998533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbd40c6-7aa3-4445-9995-b135e094f2e4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.013193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:05.036665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 317/632] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:05.037801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263ed451-e392-47ba-abdb-6507c382f2c7 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.049555 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9924c673-0718-421c-9e77-716f2b32b63b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.050220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9924c673-0718-421c-9e77-716f2b32b63b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.050493 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9924c673-0718-421c-9e77-716f2b32b63b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.050759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9924c673-0718-421c-9e77-716f2b32b63b None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.051403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9924c673-0718-421c-9e77-716f2b32b63b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:05.051701 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9924c673-0718-421c-9e77-716f2b32b63b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:05.053105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9924c673-0718-421c-9e77-716f2b32b63b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:05.071781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 316/633] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:05.072781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9924c673-0718-421c-9e77-716f2b32b63b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.161895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:05.166691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263ed451-e392-47ba-abdb-6507c382f2c7 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.203015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9924c673-0718-421c-9e77-716f2b32b63b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.204650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff80c3b8-2cfb-4998-8ade-1f2731e7de0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.206149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80c3b8-2cfb-4998-8ade-1f2731e7de0c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.206486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80c3b8-2cfb-4998-8ade-1f2731e7de0c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.206758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff80c3b8-2cfb-4998-8ade-1f2731e7de0c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.219055 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff80c3b8-2cfb-4998-8ade-1f2731e7de0c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:05.220863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff80c3b8-2cfb-4998-8ade-1f2731e7de0c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:05.236210 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ee2f24-7df5-472b-b555-092575549d90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.236811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ee2f24-7df5-472b-b555-092575549d90 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.237088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ee2f24-7df5-472b-b555-092575549d90 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.237355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ee2f24-7df5-472b-b555-092575549d90 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-ff80c3b8-2cfb-4998-8ade-1f2731e7de0c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:05.237792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:05.238853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:05.240245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:05.240245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:05.240245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:05.240430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ee2f24-7df5-472b-b555-092575549d90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:05.240430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ee2f24-7df5-472b-b555-092575549d90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:05.240555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 318/634] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:05.240555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff80c3b8-2cfb-4998-8ade-1f2731e7de0c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.244108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ee2f24-7df5-472b-b555-092575549d90 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:05.261194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 317/635] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:05.261194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02acdcb0-e4e8-483e-97ea-9d505c1637aa None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.261194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02acdcb0-e4e8-483e-97ea-9d505c1637aa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.261194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02acdcb0-e4e8-483e-97ea-9d505c1637aa None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.261194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 318/636] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:05.261194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02acdcb0-e4e8-483e-97ea-9d505c1637aa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.295314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:05.356052 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff80c3b8-2cfb-4998-8ade-1f2731e7de0c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.362634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02acdcb0-e4e8-483e-97ea-9d505c1637aa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.388755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1af84f-1bfa-4c6e-9ac4-aa8ef609b9c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.389372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1af84f-1bfa-4c6e-9ac4-aa8ef609b9c9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.389649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1af84f-1bfa-4c6e-9ac4-aa8ef609b9c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.389921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1af84f-1bfa-4c6e-9ac4-aa8ef609b9c9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.396429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1af84f-1bfa-4c6e-9ac4-aa8ef609b9c9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:05.397746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ffa13b-0afe-4bab-9ddd-e50aa3cdd04e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.397875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1af84f-1bfa-4c6e-9ac4-aa8ef609b9c9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:05.398377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ffa13b-0afe-4bab-9ddd-e50aa3cdd04e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.398654 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ffa13b-0afe-4bab-9ddd-e50aa3cdd04e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.398922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ffa13b-0afe-4bab-9ddd-e50aa3cdd04e None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.399555 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ffa13b-0afe-4bab-9ddd-e50aa3cdd04e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:05.399848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ffa13b-0afe-4bab-9ddd-e50aa3cdd04e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:05.401176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ffa13b-0afe-4bab-9ddd-e50aa3cdd04e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-0b1af84f-1bfa-4c6e-9ac4-aa8ef609b9c9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:05.408808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:05.409840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:05.411172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:05.411172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:05.411172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:05.411172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:05.411172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:05.411172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:05.411172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:05.411172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:05.411172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 319/637] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:05.411172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1af84f-1bfa-4c6e-9ac4-aa8ef609b9c9 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.419181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 319/638] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:05.419181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e333606b-fdfe-4363-ba62-f2f9bec6967d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.419181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e333606b-fdfe-4363-ba62-f2f9bec6967d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.419181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e333606b-fdfe-4363-ba62-f2f9bec6967d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.426640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:05.434317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e333606b-fdfe-4363-ba62-f2f9bec6967d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:05.435490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e333606b-fdfe-4363-ba62-f2f9bec6967d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:05.498565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 320/639] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:05.499368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e333606b-fdfe-4363-ba62-f2f9bec6967d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.519416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1af84f-1bfa-4c6e-9ac4-aa8ef609b9c9 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.527203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:05.556597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba9cba0-3586-48b8-acf5-cd1bd6bc5594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.557150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba9cba0-3586-48b8-acf5-cd1bd6bc5594 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.557426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba9cba0-3586-48b8-acf5-cd1bd6bc5594 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.557696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba9cba0-3586-48b8-acf5-cd1bd6bc5594 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.565187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba9cba0-3586-48b8-acf5-cd1bd6bc5594 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:05.566629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba9cba0-3586-48b8-acf5-cd1bd6bc5594 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-6ba9cba0-3586-48b8-acf5-cd1bd6bc5594 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:05.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:05.579194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 320/640] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/users/glance => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:05.580113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba9cba0-3586-48b8-acf5-cd1bd6bc5594 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.612957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e333606b-fdfe-4363-ba62-f2f9bec6967d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.649750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034e27fd-ff3c-46d8-8698-4c983193f41f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.650502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034e27fd-ff3c-46d8-8698-4c983193f41f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.650821 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034e27fd-ff3c-46d8-8698-4c983193f41f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.651153 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034e27fd-ff3c-46d8-8698-4c983193f41f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.655416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:05.661746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034e27fd-ff3c-46d8-8698-4c983193f41f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-034e27fd-ff3c-46d8-8698-4c983193f41f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:05.667120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:05.668115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:05.668999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 321/641] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:05.669719 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034e27fd-ff3c-46d8-8698-4c983193f41f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.694956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba9cba0-3586-48b8-acf5-cd1bd6bc5594 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.728145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf8e5d5-f0e9-4077-802f-ddc9f7bb5b1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.728743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf8e5d5-f0e9-4077-802f-ddc9f7bb5b1c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.729016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf8e5d5-f0e9-4077-802f-ddc9f7bb5b1c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.729291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf8e5d5-f0e9-4077-802f-ddc9f7bb5b1c None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.729967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf8e5d5-f0e9-4077-802f-ddc9f7bb5b1c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:05.730294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf8e5d5-f0e9-4077-802f-ddc9f7bb5b1c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:05.731705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf8e5d5-f0e9-4077-802f-ddc9f7bb5b1c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:05.749425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 321/642] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:05.750567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf8e5d5-f0e9-4077-802f-ddc9f7bb5b1c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.754676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:05.771033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034e27fd-ff3c-46d8-8698-4c983193f41f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.804796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91c0096-03d0-43d3-9e17-1d27099a4f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.805407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c0096-03d0-43d3-9e17-1d27099a4f2a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.805681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c0096-03d0-43d3-9e17-1d27099a4f2a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.805971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c0096-03d0-43d3-9e17-1d27099a4f2a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.812364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91c0096-03d0-43d3-9e17-1d27099a4f2a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:05.813728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91c0096-03d0-43d3-9e17-1d27099a4f2a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d91c0096-03d0-43d3-9e17-1d27099a4f2a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:05.824592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:05.825886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:05.826968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:05.826968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:05.826968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:05.826968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:05.826968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:05.826968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:05.826968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:05.826968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:05.826968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 322/643] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:05.826968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91c0096-03d0-43d3-9e17-1d27099a4f2a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.850648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf8e5d5-f0e9-4077-802f-ddc9f7bb5b1c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.877967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:05.884856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48948d5-10a5-4aee-b8bf-e426b892b3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.885606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48948d5-10a5-4aee-b8bf-e426b892b3e1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.886213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48948d5-10a5-4aee-b8bf-e426b892b3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.886489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48948d5-10a5-4aee-b8bf-e426b892b3e1 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.887203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48948d5-10a5-4aee-b8bf-e426b892b3e1 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:05.887506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48948d5-10a5-4aee-b8bf-e426b892b3e1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:05.888932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48948d5-10a5-4aee-b8bf-e426b892b3e1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:05.910882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 322/644] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:05.910882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48948d5-10a5-4aee-b8bf-e426b892b3e1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.926343 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91c0096-03d0-43d3-9e17-1d27099a4f2a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:05.962320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f48870-5f6d-4a09-9139-dc98b3767fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:05.962920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f48870-5f6d-4a09-9139-dc98b3767fee None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:05.963197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f48870-5f6d-4a09-9139-dc98b3767fee None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:05.963466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f48870-5f6d-4a09-9139-dc98b3767fee None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:05.964122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f48870-5f6d-4a09-9139-dc98b3767fee None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:05.964423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f48870-5f6d-4a09-9139-dc98b3767fee None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:05.965788 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f48870-5f6d-4a09-9139-dc98b3767fee None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:05.973640 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 323/645] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:05.974523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f48870-5f6d-4a09-9139-dc98b3767fee None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:05.983400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:06.004262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48948d5-10a5-4aee-b8bf-e426b892b3e1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:06.042655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d326c6-cb13-45b3-897c-656fb7467c74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:06.043257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d326c6-cb13-45b3-897c-656fb7467c74 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:06.043534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d326c6-cb13-45b3-897c-656fb7467c74 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:06.043809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d326c6-cb13-45b3-897c-656fb7467c74 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:06.044553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d326c6-cb13-45b3-897c-656fb7467c74 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=dad93d99b865407aa120035015f4692a) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:06.044857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d326c6-cb13-45b3-897c-656fb7467c74 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:06.046305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d326c6-cb13-45b3-897c-656fb7467c74 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:06.051865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 323/646] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/role_assignments?user.id=dad93d99b865407aa120035015f4692a&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:06.052653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d326c6-cb13-45b3-897c-656fb7467c74 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:06.077954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f48870-5f6d-4a09-9139-dc98b3767fee None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:06.114299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:06.115156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac2346b-e283-4dc3-9354-ceb4c72a1f74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:06.115884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac2346b-e283-4dc3-9354-ceb4c72a1f74 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:06.116291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac2346b-e283-4dc3-9354-ceb4c72a1f74 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:06.116723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac2346b-e283-4dc3-9354-ceb4c72a1f74 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:06.130333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac2346b-e283-4dc3-9354-ceb4c72a1f74 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-aac2346b-e283-4dc3-9354-ceb4c72a1f74 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:06.139474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:06.140696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:06.141736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:06.146112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 324/647] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:06.146112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac2346b-e283-4dc3-9354-ceb4c72a1f74 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:06.174148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d326c6-cb13-45b3-897c-656fb7467c74 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:06.212404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9397e652-d351-442d-afc2-158a924225a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:06.213019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9397e652-d351-442d-afc2-158a924225a7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:06.213294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9397e652-d351-442d-afc2-158a924225a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:06.213561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9397e652-d351-442d-afc2-158a924225a7 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:06.214296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9397e652-d351-442d-afc2-158a924225a7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:06.214606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9397e652-d351-442d-afc2-158a924225a7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:06.216022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9397e652-d351-442d-afc2-158a924225a7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:06.217576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:06.231129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 324/648] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:06.231312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9397e652-d351-442d-afc2-158a924225a7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:06.259749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac2346b-e283-4dc3-9354-ceb4c72a1f74 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:06.299869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5081c86-2b94-4062-920a-2a7256a48abc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:06.300486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5081c86-2b94-4062-920a-2a7256a48abc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:06.300764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5081c86-2b94-4062-920a-2a7256a48abc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:06.301038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5081c86-2b94-4062-920a-2a7256a48abc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:06.340518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9397e652-d351-442d-afc2-158a924225a7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:06.354509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:06.374378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5081c86-2b94-4062-920a-2a7256a48abc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:06.375820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5081c86-2b94-4062-920a-2a7256a48abc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:06.379795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5561b63d-2c58-4299-8a2e-7fa9687d8d89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:06.380424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5561b63d-2c58-4299-8a2e-7fa9687d8d89 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:06.380703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5561b63d-2c58-4299-8a2e-7fa9687d8d89 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:06.380977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5561b63d-2c58-4299-8a2e-7fa9687d8d89 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:06.393765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5561b63d-2c58-4299-8a2e-7fa9687d8d89 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:06.395240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5561b63d-2c58-4299-8a2e-7fa9687d8d89 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-5561b63d-2c58-4299-8a2e-7fa9687d8d89 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:06.412037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:06.413285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:06.414620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:06.414620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:06.414620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:06.414620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 325/649] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:06.414620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5561b63d-2c58-4299-8a2e-7fa9687d8d89 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:06.448675 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:06.489867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 325/650] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:42:06.489867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5081c86-2b94-4062-920a-2a7256a48abc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:06.526997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5561b63d-2c58-4299-8a2e-7fa9687d8d89 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:06.563965 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bb0e04-ccd5-4720-91cd-a00e8665f227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:06.564563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bb0e04-ccd5-4720-91cd-a00e8665f227 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:06.564837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bb0e04-ccd5-4720-91cd-a00e8665f227 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:06.565107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bb0e04-ccd5-4720-91cd-a00e8665f227 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:06.565756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bb0e04-ccd5-4720-91cd-a00e8665f227 None admin] RBAC: Adding query filter params (name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:06.566054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bb0e04-ccd5-4720-91cd-a00e8665f227 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:06.567415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bb0e04-ccd5-4720-91cd-a00e8665f227 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:06.576678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 326/651] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:42:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:06.581751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bb0e04-ccd5-4720-91cd-a00e8665f227 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:06.593315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5081c86-2b94-4062-920a-2a7256a48abc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:06.614520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:06.630421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e054b8f6-c94a-460d-85f2-c5b59fc4ac9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:06.631383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e054b8f6-c94a-460d-85f2-c5b59fc4ac9b None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:06.631948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e054b8f6-c94a-460d-85f2-c5b59fc4ac9b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:06.632485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e054b8f6-c94a-460d-85f2-c5b59fc4ac9b None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/db6c9dbbcf244a6f8bac59d2c9c973bc/roles/2c1bcaa088914d4593bb1debe2fc88d6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:06.648232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e054b8f6-c94a-460d-85f2-c5b59fc4ac9b None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=db6c9dbbcf244a6f8bac59d2c9c973bc, role_id=2c1bcaa088914d4593bb1debe2fc88d6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:06.649822 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e054b8f6-c94a-460d-85f2-c5b59fc4ac9b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:06.674161 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 326/652] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:42:06 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/db6c9dbbcf244a6f8bac59d2c9c973bc/roles/2c1bcaa088914d4593bb1debe2fc88d6 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:42:06.675198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e054b8f6-c94a-460d-85f2-c5b59fc4ac9b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:06.695769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bb0e04-ccd5-4720-91cd-a00e8665f227 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:06.705984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:06.738848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aff5e64-b314-4fb0-9bd1-b059dc99b829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:06.739902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aff5e64-b314-4fb0-9bd1-b059dc99b829 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:06.740332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aff5e64-b314-4fb0-9bd1-b059dc99b829 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:06.740786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aff5e64-b314-4fb0-9bd1-b059dc99b829 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:06.742551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aff5e64-b314-4fb0-9bd1-b059dc99b829 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:06.742864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aff5e64-b314-4fb0-9bd1-b059dc99b829 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:06.744378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aff5e64-b314-4fb0-9bd1-b059dc99b829 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:06.760919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 327/653] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:06.762193 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aff5e64-b314-4fb0-9bd1-b059dc99b829 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:06.810518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e054b8f6-c94a-460d-85f2-c5b59fc4ac9b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:06.858623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ae6443-f6b2-402b-aaeb-f23aebb93523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:06.860194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ae6443-f6b2-402b-aaeb-f23aebb93523 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:06.860673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ae6443-f6b2-402b-aaeb-f23aebb93523 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:06.861202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ae6443-f6b2-402b-aaeb-f23aebb93523 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:06.864419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:06.876052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ae6443-f6b2-402b-aaeb-f23aebb93523 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:06.877997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ae6443-f6b2-402b-aaeb-f23aebb93523 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:06.883743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aff5e64-b314-4fb0-9bd1-b059dc99b829 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b4ae6443-f6b2-402b-aaeb-f23aebb93523 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:06.898690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:06.899895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:06.900981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:06.900981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:06.900981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:06.902729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 327/654] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:06.903764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ae6443-f6b2-402b-aaeb-f23aebb93523 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:06.926655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500285b2-31d5-4cfe-8be9-551cbd35fc8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:06.927433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500285b2-31d5-4cfe-8be9-551cbd35fc8c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:06.927806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500285b2-31d5-4cfe-8be9-551cbd35fc8c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:06.928192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500285b2-31d5-4cfe-8be9-551cbd35fc8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:06.940190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500285b2-31d5-4cfe-8be9-551cbd35fc8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:06.941990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500285b2-31d5-4cfe-8be9-551cbd35fc8c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-500285b2-31d5-4cfe-8be9-551cbd35fc8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:06.959246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:06.961271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:06.962188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:06.962188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:06.962188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:06.962188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 328/655] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:06.963012 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500285b2-31d5-4cfe-8be9-551cbd35fc8c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:07.001998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:07.053070 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ae6443-f6b2-402b-aaeb-f23aebb93523 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:07.088731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500285b2-31d5-4cfe-8be9-551cbd35fc8c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:07.090601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb25ad9-a6fe-45f1-92d4-ac5fc9a48ebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:07.091561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb25ad9-a6fe-45f1-92d4-ac5fc9a48ebb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:07.092044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb25ad9-a6fe-45f1-92d4-ac5fc9a48ebb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:07.092532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb25ad9-a6fe-45f1-92d4-ac5fc9a48ebb None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:07.100921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb25ad9-a6fe-45f1-92d4-ac5fc9a48ebb None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:07.102688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb25ad9-a6fe-45f1-92d4-ac5fc9a48ebb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5bb25ad9-a6fe-45f1-92d4-ac5fc9a48ebb None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:07.114867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:07.115754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:07.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:07.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:07.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:07.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:07.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:07.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:07.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:07.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:07.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 02 00:42:07.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:07.119955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 328/656] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:42:06 2022] GET /identity/v3/users/cinder => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:07.120941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb25ad9-a6fe-45f1-92d4-ac5fc9a48ebb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:07.124196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cafeba-b4dc-4b4a-b415-f9ad95295392 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:07.125618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cafeba-b4dc-4b4a-b415-f9ad95295392 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:07.125903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cafeba-b4dc-4b4a-b415-f9ad95295392 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:07.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cafeba-b4dc-4b4a-b415-f9ad95295392 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:07.126848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cafeba-b4dc-4b4a-b415-f9ad95295392 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:07.127148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cafeba-b4dc-4b4a-b415-f9ad95295392 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:07.128490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cafeba-b4dc-4b4a-b415-f9ad95295392 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:07.137243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:07.146129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 329/657] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:07.147215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cafeba-b4dc-4b4a-b415-f9ad95295392 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:07.252658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb25ad9-a6fe-45f1-92d4-ac5fc9a48ebb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:07.268989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cafeba-b4dc-4b4a-b415-f9ad95295392 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:07.283540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:07.296776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b7265-a20b-45bf-a2d8-ad4940ec2fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:07.298099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b7265-a20b-45bf-a2d8-ad4940ec2fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:07.298807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b7265-a20b-45bf-a2d8-ad4940ec2fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:07.299549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b7265-a20b-45bf-a2d8-ad4940ec2fc3 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:07.300849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75b7265-a20b-45bf-a2d8-ad4940ec2fc3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:07.301627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75b7265-a20b-45bf-a2d8-ad4940ec2fc3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:07.304900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75b7265-a20b-45bf-a2d8-ad4940ec2fc3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:07.308188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd190e87-fb44-433c-ac36-ba056e4b8a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:07.308799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd190e87-fb44-433c-ac36-ba056e4b8a36 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:07.309078 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd190e87-fb44-433c-ac36-ba056e4b8a36 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:07.309349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd190e87-fb44-433c-ac36-ba056e4b8a36 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:07.310115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd190e87-fb44-433c-ac36-ba056e4b8a36 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:07.310401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd190e87-fb44-433c-ac36-ba056e4b8a36 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:07.312312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd190e87-fb44-433c-ac36-ba056e4b8a36 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:07.324056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 329/658] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:07.325226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b7265-a20b-45bf-a2d8-ad4940ec2fc3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:07.333299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 330/659] 10.222.0.25 () {56 vars in 1137 bytes} [Sun Oct 2 00:42:07 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:07.334291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd190e87-fb44-433c-ac36-ba056e4b8a36 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:07.399761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:07.451594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd190e87-fb44-433c-ac36-ba056e4b8a36 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:07.466400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b7265-a20b-45bf-a2d8-ad4940ec2fc3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:07.495661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e118d1-271b-421b-89ef-e775a4575261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:07.496677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e118d1-271b-421b-89ef-e775a4575261 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:07.497425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e118d1-271b-421b-89ef-e775a4575261 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:07.498166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e118d1-271b-421b-89ef-e775a4575261 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:07.502035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df394426-5905-490c-8a25-60c53bda78a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:07.502686 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df394426-5905-490c-8a25-60c53bda78a4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:07.502964 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df394426-5905-490c-8a25-60c53bda78a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:07.503236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df394426-5905-490c-8a25-60c53bda78a4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:07.507240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e118d1-271b-421b-89ef-e775a4575261 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:07.509115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e118d1-271b-421b-89ef-e775a4575261 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:07.511845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df394426-5905-490c-8a25-60c53bda78a4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:07.513264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df394426-5905-490c-8a25-60c53bda78a4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-75e118d1-271b-421b-89ef-e775a4575261 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:07.525199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:07.526396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 331/660] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:07 2022] GET /identity/v3/users/neutron => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:07.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e118d1-271b-421b-89ef-e775a4575261 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-df394426-5905-490c-8a25-60c53bda78a4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:07.532112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:07.532999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:07.534197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:07.534197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:07.534197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:07.534197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:07.534197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:07.534197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:07.534197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:07.534197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:07.536151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 330/661] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:07.542413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df394426-5905-490c-8a25-60c53bda78a4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:07.605573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:07.666117 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e118d1-271b-421b-89ef-e775a4575261 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:07.670989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df394426-5905-490c-8a25-60c53bda78a4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:07.685070 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:07.706606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19552ac-7b08-43f3-b1af-a95e8f946d7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:07.707206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19552ac-7b08-43f3-b1af-a95e8f946d7d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:07.707473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19552ac-7b08-43f3-b1af-a95e8f946d7d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:07.707740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19552ac-7b08-43f3-b1af-a95e8f946d7d None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:07.708278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433a30bc-e080-4930-814b-fa61286d7573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:07.708450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19552ac-7b08-43f3-b1af-a95e8f946d7d None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:07.708750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19552ac-7b08-43f3-b1af-a95e8f946d7d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:07.709317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433a30bc-e080-4930-814b-fa61286d7573 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:07.709777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433a30bc-e080-4930-814b-fa61286d7573 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:07.710265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19552ac-7b08-43f3-b1af-a95e8f946d7d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:07.710445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433a30bc-e080-4930-814b-fa61286d7573 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:07.733603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 331/662] 10.222.0.25 () {56 vars in 1139 bytes} [Sun Oct 2 00:42:07 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:07.734515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19552ac-7b08-43f3-b1af-a95e8f946d7d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:07.788462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433a30bc-e080-4930-814b-fa61286d7573 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:07.790294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433a30bc-e080-4930-814b-fa61286d7573 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:07.839545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19552ac-7b08-43f3-b1af-a95e8f946d7d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:07.847894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:07.878203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a8a5a6-d9cb-49c2-be58-f0ff828054e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:07.878833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a8a5a6-d9cb-49c2-be58-f0ff828054e3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:07.879061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a8a5a6-d9cb-49c2-be58-f0ff828054e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:07.879325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a8a5a6-d9cb-49c2-be58-f0ff828054e3 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:07.880006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a8a5a6-d9cb-49c2-be58-f0ff828054e3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:07.880305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a8a5a6-d9cb-49c2-be58-f0ff828054e3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:07.881783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a8a5a6-d9cb-49c2-be58-f0ff828054e3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:07.899743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 332/663] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:07.900710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a8a5a6-d9cb-49c2-be58-f0ff828054e3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:07.913109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 332/664] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:42:07.914626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433a30bc-e080-4930-814b-fa61286d7573 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:07.935843 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:08.027262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a8a5a6-d9cb-49c2-be58-f0ff828054e3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:08.045913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433a30bc-e080-4930-814b-fa61286d7573 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:08.068433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc98683-345b-4d60-9bcc-d837c5865c95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:08.069174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc98683-345b-4d60-9bcc-d837c5865c95 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:08.069572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc98683-345b-4d60-9bcc-d837c5865c95 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:08.069854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc98683-345b-4d60-9bcc-d837c5865c95 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:08.091307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb206aff-c9f6-4254-a70b-4bbb9be13ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:08.091969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb206aff-c9f6-4254-a70b-4bbb9be13ce0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:08.092380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb206aff-c9f6-4254-a70b-4bbb9be13ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:08.092653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb206aff-c9f6-4254-a70b-4bbb9be13ce0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:08.093907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb206aff-c9f6-4254-a70b-4bbb9be13ce0 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=30fc59f80f544cef8b3869e8d37bb341) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:08.094246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb206aff-c9f6-4254-a70b-4bbb9be13ce0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:08.096217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb206aff-c9f6-4254-a70b-4bbb9be13ce0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:08.105001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 333/665] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:07 2022] GET /identity/v3/role_assignments?user.id=30fc59f80f544cef8b3869e8d37bb341&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:08.105922 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb206aff-c9f6-4254-a70b-4bbb9be13ce0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:08.106469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:08.196805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc98683-345b-4d60-9bcc-d837c5865c95 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:08.198118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc98683-345b-4d60-9bcc-d837c5865c95 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:08.210217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:08.256021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb206aff-c9f6-4254-a70b-4bbb9be13ce0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:08.311202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52a5a27-f78e-482f-8b66-86d9cb8f5cbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:08.312801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52a5a27-f78e-482f-8b66-86d9cb8f5cbc None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:08.313235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52a5a27-f78e-482f-8b66-86d9cb8f5cbc None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:08.313651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52a5a27-f78e-482f-8b66-86d9cb8f5cbc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:08.327018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52a5a27-f78e-482f-8b66-86d9cb8f5cbc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:08.329526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52a5a27-f78e-482f-8b66-86d9cb8f5cbc None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-d52a5a27-f78e-482f-8b66-86d9cb8f5cbc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:08.347235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:08.348227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:08.349576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:08.349576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:08.349576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:08.349576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 334/666] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:08.356410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52a5a27-f78e-482f-8b66-86d9cb8f5cbc None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:08.361788 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 333/667] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:42:08.363273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8760664-3805-4c7a-939a-e69e6b9d8a25 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:08.363746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8760664-3805-4c7a-939a-e69e6b9d8a25 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:08.364174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8760664-3805-4c7a-939a-e69e6b9d8a25 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:08.366032 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 334/668] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:08.367013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8760664-3805-4c7a-939a-e69e6b9d8a25 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:08.445895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:08.486606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8760664-3805-4c7a-939a-e69e6b9d8a25 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:08.491289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52a5a27-f78e-482f-8b66-86d9cb8f5cbc None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:08.531695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f994e1ed-31cb-4b52-b3dd-1028e27fc891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:08.532174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe8e9be-8c2b-4374-b7d7-c915d8308fab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:08.532347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f994e1ed-31cb-4b52-b3dd-1028e27fc891 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:08.532639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f994e1ed-31cb-4b52-b3dd-1028e27fc891 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:08.532911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f994e1ed-31cb-4b52-b3dd-1028e27fc891 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:08.533025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe8e9be-8c2b-4374-b7d7-c915d8308fab None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:08.533313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe8e9be-8c2b-4374-b7d7-c915d8308fab None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:08.533594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe8e9be-8c2b-4374-b7d7-c915d8308fab None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:08.534284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe8e9be-8c2b-4374-b7d7-c915d8308fab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:08.534588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe8e9be-8c2b-4374-b7d7-c915d8308fab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:08.535963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe8e9be-8c2b-4374-b7d7-c915d8308fab None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:08.543856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:08.545741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f994e1ed-31cb-4b52-b3dd-1028e27fc891 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:08.548070 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f994e1ed-31cb-4b52-b3dd-1028e27fc891 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:08.549966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 335/669] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:08.551676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1423aff9-0527-42bd-bee4-0aeb15299122 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:08.552104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1423aff9-0527-42bd-bee4-0aeb15299122 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:08.552472 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1423aff9-0527-42bd-bee4-0aeb15299122 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-f994e1ed-31cb-4b52-b3dd-1028e27fc891 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:08.570008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:08.571249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:08.572319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:08.572319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:08.572319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:08.572319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 335/670] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:08.572319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f994e1ed-31cb-4b52-b3dd-1028e27fc891 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:08.578156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-1423aff9-0527-42bd-bee4-0aeb15299122 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:08.579701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1423aff9-0527-42bd-bee4-0aeb15299122 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:08.662217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 336/671] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:08.663172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1423aff9-0527-42bd-bee4-0aeb15299122 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:08.725879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f994e1ed-31cb-4b52-b3dd-1028e27fc891 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:08.762424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:08.784424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0842aede-df36-46f5-b361-e79550096f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:08.785058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0842aede-df36-46f5-b361-e79550096f18 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:08.785336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0842aede-df36-46f5-b361-e79550096f18 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:08.785615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0842aede-df36-46f5-b361-e79550096f18 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:08.793204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0842aede-df36-46f5-b361-e79550096f18 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:08.794672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0842aede-df36-46f5-b361-e79550096f18 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0842aede-df36-46f5-b361-e79550096f18 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:08.808301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:08.809228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:08.810671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:08.810671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:08.810671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:08.810671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:08.810671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:08.810671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:08.810671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:08.810671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:08.810671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 336/672] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:08.817386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0842aede-df36-46f5-b361-e79550096f18 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:08.850040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1423aff9-0527-42bd-bee4-0aeb15299122 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:08.901311 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:08.932235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d55e74-5ed0-4e72-9253-ba31a9124354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:08.933650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d55e74-5ed0-4e72-9253-ba31a9124354 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:08.934211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d55e74-5ed0-4e72-9253-ba31a9124354 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:08.934810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d55e74-5ed0-4e72-9253-ba31a9124354 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:08.935661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d55e74-5ed0-4e72-9253-ba31a9124354 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:08.936174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d55e74-5ed0-4e72-9253-ba31a9124354 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:08.937975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d55e74-5ed0-4e72-9253-ba31a9124354 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:08.948079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0842aede-df36-46f5-b361-e79550096f18 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:08.963413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 337/673] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:08.969728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d55e74-5ed0-4e72-9253-ba31a9124354 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:08.988457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd7b27e-97b6-4e07-b649-77c873aa0801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:08.989282 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd7b27e-97b6-4e07-b649-77c873aa0801 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:08.990027 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd7b27e-97b6-4e07-b649-77c873aa0801 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:08.990337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd7b27e-97b6-4e07-b649-77c873aa0801 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:08.991510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd7b27e-97b6-4e07-b649-77c873aa0801 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:08.991862 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd7b27e-97b6-4e07-b649-77c873aa0801 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:08.994024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd7b27e-97b6-4e07-b649-77c873aa0801 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:09.011222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 337/674] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:09.014227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd7b27e-97b6-4e07-b649-77c873aa0801 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:09.058011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:09.095033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d55e74-5ed0-4e72-9253-ba31a9124354 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:09.123012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd7b27e-97b6-4e07-b649-77c873aa0801 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:09.127651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f47473-98ca-43d4-adac-e546bfc824ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:09.128275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f47473-98ca-43d4-adac-e546bfc824ff None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:09.128547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f47473-98ca-43d4-adac-e546bfc824ff None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:09.128812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f47473-98ca-43d4-adac-e546bfc824ff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:09.136276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f47473-98ca-43d4-adac-e546bfc824ff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:09.137697 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f47473-98ca-43d4-adac-e546bfc824ff None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-f3f47473-98ca-43d4-adac-e546bfc824ff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:09.149164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:09.150049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:09.151017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:09.151017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:09.151017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:09.151017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:09.151017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:09.151017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:09.151017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:09.151017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:09.151017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 338/675] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:09.151495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f47473-98ca-43d4-adac-e546bfc824ff None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:09.164143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4705bfda-7bae-497f-91cb-62855eab7b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:09.164773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4705bfda-7bae-497f-91cb-62855eab7b15 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:09.165110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4705bfda-7bae-497f-91cb-62855eab7b15 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:09.165381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4705bfda-7bae-497f-91cb-62855eab7b15 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:09.178427 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4705bfda-7bae-497f-91cb-62855eab7b15 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4705bfda-7bae-497f-91cb-62855eab7b15 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:09.183237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:09.184179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:09.185017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 338/676] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:09.185703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77edebfb-3f4f-44d7-a3bc-6aae6ceca846 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:09.185703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77edebfb-3f4f-44d7-a3bc-6aae6ceca846 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:09.186520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77edebfb-3f4f-44d7-a3bc-6aae6ceca846 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:09.190150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 339/677] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:09 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:09.190150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77edebfb-3f4f-44d7-a3bc-6aae6ceca846 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:09.206278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:09.265814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f47473-98ca-43d4-adac-e546bfc824ff None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:09.294329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:09.300747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77edebfb-3f4f-44d7-a3bc-6aae6ceca846 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:09.303702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc46b43-f8c2-4c7b-bf52-b02463cdd236 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:09.304318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc46b43-f8c2-4c7b-bf52-b02463cdd236 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:09.304599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc46b43-f8c2-4c7b-bf52-b02463cdd236 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:09.304871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc46b43-f8c2-4c7b-bf52-b02463cdd236 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:09.305602 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc46b43-f8c2-4c7b-bf52-b02463cdd236 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=eb325d3c277b45c2a5c4e0e94078be29) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:09.305905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc46b43-f8c2-4c7b-bf52-b02463cdd236 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:09.307361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc46b43-f8c2-4c7b-bf52-b02463cdd236 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:09.313691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 339/678] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:09 2022] GET /identity/v3/role_assignments?user.id=eb325d3c277b45c2a5c4e0e94078be29&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:09.314503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc46b43-f8c2-4c7b-bf52-b02463cdd236 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:09.349891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d9bf45-66a6-45cd-9de1-4825131ca379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:09.350647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d9bf45-66a6-45cd-9de1-4825131ca379 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:09.350922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d9bf45-66a6-45cd-9de1-4825131ca379 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:09.351191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d9bf45-66a6-45cd-9de1-4825131ca379 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:09.351881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d9bf45-66a6-45cd-9de1-4825131ca379 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:09.352184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d9bf45-66a6-45cd-9de1-4825131ca379 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:09.353797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d9bf45-66a6-45cd-9de1-4825131ca379 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:09.371852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 340/679] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:09.373108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f45179-e1a3-4393-8b20-a64a7e1a4d58 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:09.373385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f45179-e1a3-4393-8b20-a64a7e1a4d58 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:09.373652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f45179-e1a3-4393-8b20-a64a7e1a4d58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:09.394783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-63f45179-e1a3-4393-8b20-a64a7e1a4d58 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:09.396362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-63f45179-e1a3-4393-8b20-a64a7e1a4d58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:09.440126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc46b43-f8c2-4c7b-bf52-b02463cdd236 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:09.448194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:09.480196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb8de8e-6759-4312-b231-6b9dbc00de28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:09.480806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb8de8e-6759-4312-b231-6b9dbc00de28 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:09.481079 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb8de8e-6759-4312-b231-6b9dbc00de28 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:09.481345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb8de8e-6759-4312-b231-6b9dbc00de28 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:09.482006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb8de8e-6759-4312-b231-6b9dbc00de28 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:09.482355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb8de8e-6759-4312-b231-6b9dbc00de28 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:09.483842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb8de8e-6759-4312-b231-6b9dbc00de28 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:09.487291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 341/680] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:09.488485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f45179-e1a3-4393-8b20-a64a7e1a4d58 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:09.498361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 340/681] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:42:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:09.499855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb8de8e-6759-4312-b231-6b9dbc00de28 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:09.606903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:09.618090 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb8de8e-6759-4312-b231-6b9dbc00de28 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:09.620899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f45179-e1a3-4393-8b20-a64a7e1a4d58 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:09.654603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89441183-31f7-4f81-b9e0-a6609bef6d45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:09.655229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89441183-31f7-4f81-b9e0-a6609bef6d45 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:09.655501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89441183-31f7-4f81-b9e0-a6609bef6d45 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:09.655771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89441183-31f7-4f81-b9e0-a6609bef6d45 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:09.665162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6eca85e-766c-4002-b2ad-31ea95d05a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:09.665778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eca85e-766c-4002-b2ad-31ea95d05a83 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:09.666146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eca85e-766c-4002-b2ad-31ea95d05a83 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:09.666426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eca85e-766c-4002-b2ad-31ea95d05a83 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:09.667216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6eca85e-766c-4002-b2ad-31ea95d05a83 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=e75811ee9ed7490985fed336ab9feb19) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:09.667327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89441183-31f7-4f81-b9e0-a6609bef6d45 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:09.667528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6eca85e-766c-4002-b2ad-31ea95d05a83 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:09.668646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89441183-31f7-4f81-b9e0-a6609bef6d45 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:09.669281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6eca85e-766c-4002-b2ad-31ea95d05a83 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:09.677601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 342/682] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:09 2022] GET /identity/v3/role_assignments?user.id=e75811ee9ed7490985fed336ab9feb19&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-89441183-31f7-4f81-b9e0-a6609bef6d45 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:09.686890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:09.687992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:09.688886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:09.688886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:09.688886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:09.688886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 341/683] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:09.694773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6eca85e-766c-4002-b2ad-31ea95d05a83 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:09.804753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89441183-31f7-4f81-b9e0-a6609bef6d45 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:09.821285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6eca85e-766c-4002-b2ad-31ea95d05a83 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:09.831520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:09.890355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061750f5-91ff-4a12-9639-e50f8d0a8293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:09.890895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061750f5-91ff-4a12-9639-e50f8d0a8293 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:09.891173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061750f5-91ff-4a12-9639-e50f8d0a8293 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:09.891441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061750f5-91ff-4a12-9639-e50f8d0a8293 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:09.892090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061750f5-91ff-4a12-9639-e50f8d0a8293 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:09.892388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061750f5-91ff-4a12-9639-e50f8d0a8293 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:09.893789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061750f5-91ff-4a12-9639-e50f8d0a8293 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:09.910912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 343/684] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:09.917857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061750f5-91ff-4a12-9639-e50f8d0a8293 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:09.936154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:09.962691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89441183-31f7-4f81-b9e0-a6609bef6d45 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:10.006033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9cde0f-056d-4568-9e11-ef262a6fb388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:10.007239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9cde0f-056d-4568-9e11-ef262a6fb388 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:10.007516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9cde0f-056d-4568-9e11-ef262a6fb388 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:10.007789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9cde0f-056d-4568-9e11-ef262a6fb388 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:10.019841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9cde0f-056d-4568-9e11-ef262a6fb388 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-4b9cde0f-056d-4568-9e11-ef262a6fb388 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:10.025423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:10.026395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:10.027286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 342/685] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:10.033652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9cde0f-056d-4568-9e11-ef262a6fb388 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:10.045105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061750f5-91ff-4a12-9639-e50f8d0a8293 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:10.082456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1d617-7808-4adb-94d0-c56b9c552ba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:10.083455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1d617-7808-4adb-94d0-c56b9c552ba9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:10.083939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1d617-7808-4adb-94d0-c56b9c552ba9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:10.084442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc1d617-7808-4adb-94d0-c56b9c552ba9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:10.093636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc1d617-7808-4adb-94d0-c56b9c552ba9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:10.096183 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc1d617-7808-4adb-94d0-c56b9c552ba9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ffc1d617-7808-4adb-94d0-c56b9c552ba9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:10.109151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:10.111131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:10.112096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:10.112096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:10.112096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:10.112096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:10.112096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:10.112096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:10.112096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:10.112096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:10.112096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:42:10.112096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:10.114450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 344/686] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 2 00:42:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:10.124254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1d617-7808-4adb-94d0-c56b9c552ba9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:10.139866 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:10.165151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9cde0f-056d-4568-9e11-ef262a6fb388 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:10.210517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:10.245233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc1d617-7808-4adb-94d0-c56b9c552ba9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:10.251324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685f518-4cce-422f-9976-216333638024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:10.251949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685f518-4cce-422f-9976-216333638024 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:10.252229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685f518-4cce-422f-9976-216333638024 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:10.252503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685f518-4cce-422f-9976-216333638024 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:10.253176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685f518-4cce-422f-9976-216333638024 None admin] RBAC: Adding query filter params (name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:10.253481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685f518-4cce-422f-9976-216333638024 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:10.254899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685f518-4cce-422f-9976-216333638024 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:10.273280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 343/687] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:42:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:10.278875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685f518-4cce-422f-9976-216333638024 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:10.284974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9f234c-2a7f-467f-9936-f06557ce3892 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:10.286006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9f234c-2a7f-467f-9936-f06557ce3892 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:10.286512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9f234c-2a7f-467f-9936-f06557ce3892 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:10.286997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9f234c-2a7f-467f-9936-f06557ce3892 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:10.288047 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9f234c-2a7f-467f-9936-f06557ce3892 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:10.288556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9f234c-2a7f-467f-9936-f06557ce3892 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:10.290282 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9f234c-2a7f-467f-9936-f06557ce3892 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:10.309647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 345/688] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 2 00:42:10 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:10.315496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9f234c-2a7f-467f-9936-f06557ce3892 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:10.443773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9f234c-2a7f-467f-9936-f06557ce3892 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:10.467501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685f518-4cce-422f-9976-216333638024 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:10.505042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a853c6-1224-416c-b7b1-02ca0b829f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:10.506022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a853c6-1224-416c-b7b1-02ca0b829f91 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:10.507178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a853c6-1224-416c-b7b1-02ca0b829f91 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:10.507738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a853c6-1224-416c-b7b1-02ca0b829f91 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:10.519544 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a853c6-1224-416c-b7b1-02ca0b829f91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:10.521480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a853c6-1224-416c-b7b1-02ca0b829f91 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:10.522490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:10.524155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-56a853c6-1224-416c-b7b1-02ca0b829f91 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:10.540044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:10.541039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:10.541943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:10.541943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:10.541943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:10.543666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 344/689] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:10.548625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7ba6a5-5034-44ad-b1df-e49e94fa0058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:10.548625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ba6a5-5034-44ad-b1df-e49e94fa0058 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:10.548625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ba6a5-5034-44ad-b1df-e49e94fa0058 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:10.548625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ba6a5-5034-44ad-b1df-e49e94fa0058 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:10.551086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a853c6-1224-416c-b7b1-02ca0b829f91 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:10.665388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a853c6-1224-416c-b7b1-02ca0b829f91 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:10.701285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1900c5e-3aaa-4b32-99bd-504fbae794d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:10.702265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1900c5e-3aaa-4b32-99bd-504fbae794d1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:10.702895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1900c5e-3aaa-4b32-99bd-504fbae794d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:10.703375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1900c5e-3aaa-4b32-99bd-504fbae794d1 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:10.704212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1900c5e-3aaa-4b32-99bd-504fbae794d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:10.704714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1900c5e-3aaa-4b32-99bd-504fbae794d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:10.706314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1900c5e-3aaa-4b32-99bd-504fbae794d1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:10.718590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7ba6a5-5034-44ad-b1df-e49e94fa0058 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:10.720042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7ba6a5-5034-44ad-b1df-e49e94fa0058 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:10.721169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 345/690] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:10.726962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa77fee-e6fb-4881-9745-3196b81067e4 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:10.727185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa77fee-e6fb-4881-9745-3196b81067e4 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:10.728319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa77fee-e6fb-4881-9745-3196b81067e4 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:10.731211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 346/691] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:10 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:10.732149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa77fee-e6fb-4881-9745-3196b81067e4 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:10.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:10.838902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 346/692] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 519 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 02 00:42:10.838902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbf23aa-c5ab-451e-aa18-c3e35c044aea None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:10.838902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbf23aa-c5ab-451e-aa18-c3e35c044aea None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:10.838902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbf23aa-c5ab-451e-aa18-c3e35c044aea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:10.854481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa77fee-e6fb-4881-9745-3196b81067e4 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:10.856660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2cbf23aa-c5ab-451e-aa18-c3e35c044aea None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:10.858233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2cbf23aa-c5ab-451e-aa18-c3e35c044aea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:10.884788 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:10.897937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f76c905-2fe4-49f5-bfdf-07d0ab6c3518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:10.899894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f76c905-2fe4-49f5-bfdf-07d0ab6c3518 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:10.900394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f76c905-2fe4-49f5-bfdf-07d0ab6c3518 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:10.900870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f76c905-2fe4-49f5-bfdf-07d0ab6c3518 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:10.910010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f76c905-2fe4-49f5-bfdf-07d0ab6c3518 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:10.911754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f76c905-2fe4-49f5-bfdf-07d0ab6c3518 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-5f76c905-2fe4-49f5-bfdf-07d0ab6c3518 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:10.926349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:10.927342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:10.928865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:10.928865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:10.928865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:10.928865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:10.928865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:10.928865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:10.928865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:10.928865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:10.928865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:42:10.928865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:10.936387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 347/693] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:10 2022] GET /identity/v3/users/swift => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:10.936387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f76c905-2fe4-49f5-bfdf-07d0ab6c3518 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:10.946589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 347/694] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:10.947722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbf23aa-c5ab-451e-aa18-c3e35c044aea None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:11.066282 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbf23aa-c5ab-451e-aa18-c3e35c044aea None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:11.107324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a08e07-4134-4622-ba8c-4d85b92a4723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:11.108575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a08e07-4134-4622-ba8c-4d85b92a4723 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:11.109055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a08e07-4134-4622-ba8c-4d85b92a4723 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:11.109788 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a08e07-4134-4622-ba8c-4d85b92a4723 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:11.110754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a08e07-4134-4622-ba8c-4d85b92a4723 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:11.111264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a08e07-4134-4622-ba8c-4d85b92a4723 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:11.112909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a08e07-4134-4622-ba8c-4d85b92a4723 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:11.135313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 348/695] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 2 00:42:10 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:11.141329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a08e07-4134-4622-ba8c-4d85b92a4723 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:11.162716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:11.191401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f76c905-2fe4-49f5-bfdf-07d0ab6c3518 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:11.220956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:11.259833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa7a299-9783-4991-8434-8af79b95ce93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:11.260462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa7a299-9783-4991-8434-8af79b95ce93 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:11.260738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa7a299-9783-4991-8434-8af79b95ce93 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:11.261004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa7a299-9783-4991-8434-8af79b95ce93 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:11.280692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa7a299-9783-4991-8434-8af79b95ce93 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:11.282164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa7a299-9783-4991-8434-8af79b95ce93 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-2aa7a299-9783-4991-8434-8af79b95ce93 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:11.309189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:11.310219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:11.311150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:11.311150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:11.311150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:11.312375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a08e07-4134-4622-ba8c-4d85b92a4723 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:11.313443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 348/696] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 381 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:11.314557 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa7a299-9783-4991-8434-8af79b95ce93 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:11.362118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71616858-e088-4ccd-b47b-ac29521940d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:11.363087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71616858-e088-4ccd-b47b-ac29521940d2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:11.363566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71616858-e088-4ccd-b47b-ac29521940d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:11.364636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71616858-e088-4ccd-b47b-ac29521940d2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:11.437689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa7a299-9783-4991-8434-8af79b95ce93 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:11.438417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71616858-e088-4ccd-b47b-ac29521940d2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:11.440078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71616858-e088-4ccd-b47b-ac29521940d2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:11.459046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:11.480332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d596ec-b98e-414a-8703-cbc00a2f85e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:11.480961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d596ec-b98e-414a-8703-cbc00a2f85e9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:11.481233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d596ec-b98e-414a-8703-cbc00a2f85e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:11.481502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d596ec-b98e-414a-8703-cbc00a2f85e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:11.494333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d596ec-b98e-414a-8703-cbc00a2f85e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:11.495778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d596ec-b98e-414a-8703-cbc00a2f85e9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-05d596ec-b98e-414a-8703-cbc00a2f85e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:11.513618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:11.514790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:11.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:11.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:11.516057 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:11.518200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 349/697] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:11.519271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d596ec-b98e-414a-8703-cbc00a2f85e9 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:11.544511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:11.555705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 349/698] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 02 00:42:11.556891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71616858-e088-4ccd-b47b-ac29521940d2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:11.652033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d596ec-b98e-414a-8703-cbc00a2f85e9 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:11.665856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71616858-e088-4ccd-b47b-ac29521940d2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:11.684305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:11.692885 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ec5a3d-1938-4371-94dc-61bea2e7e12a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:11.693631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ec5a3d-1938-4371-94dc-61bea2e7e12a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:11.694234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ec5a3d-1938-4371-94dc-61bea2e7e12a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:11.694515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ec5a3d-1938-4371-94dc-61bea2e7e12a None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:11.695155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ec5a3d-1938-4371-94dc-61bea2e7e12a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:11.695454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ec5a3d-1938-4371-94dc-61bea2e7e12a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:11.697168 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ec5a3d-1938-4371-94dc-61bea2e7e12a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:11.705018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06051618-e365-4233-af87-decd540a1e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:11.705871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06051618-e365-4233-af87-decd540a1e02 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:11.706405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06051618-e365-4233-af87-decd540a1e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:11.706881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06051618-e365-4233-af87-decd540a1e02 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:11.707714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06051618-e365-4233-af87-decd540a1e02 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:11.708230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06051618-e365-4233-af87-decd540a1e02 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:11.710326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06051618-e365-4233-af87-decd540a1e02 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:11.716179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 350/699] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:11.717066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ec5a3d-1938-4371-94dc-61bea2e7e12a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:11.723549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 350/700] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:11.724618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06051618-e365-4233-af87-decd540a1e02 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:11.819613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:11.836324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06051618-e365-4233-af87-decd540a1e02 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:11.844846 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ec5a3d-1938-4371-94dc-61bea2e7e12a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:11.874030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7ebdc8-c3de-4062-90ee-5f40fe4ff9ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:11.875102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7ebdc8-c3de-4062-90ee-5f40fe4ff9ce None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:11.875582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7ebdc8-c3de-4062-90ee-5f40fe4ff9ce None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:11.876061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7ebdc8-c3de-4062-90ee-5f40fe4ff9ce None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:11.884805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7ebdc8-c3de-4062-90ee-5f40fe4ff9ce None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:11.885608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009b89bf-055f-449a-b61e-fe664aabbef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:11.886293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009b89bf-055f-449a-b61e-fe664aabbef0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:11.886640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009b89bf-055f-449a-b61e-fe664aabbef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:11.886915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009b89bf-055f-449a-b61e-fe664aabbef0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:11.887652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7ebdc8-c3de-4062-90ee-5f40fe4ff9ce None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:11.900139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009b89bf-055f-449a-b61e-fe664aabbef0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:11.901602 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009b89bf-055f-449a-b61e-fe664aabbef0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fb7ebdc8-c3de-4062-90ee-5f40fe4ff9ce None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:11.902453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:11.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:11.904352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:11.904352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:11.904352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:11.904352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:11.904352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:11.904352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:11.904352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:11.904352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:11.906467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 351/701] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:11.907561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7ebdc8-c3de-4062-90ee-5f40fe4ff9ce None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-009b89bf-055f-449a-b61e-fe664aabbef0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:11.922202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:11.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:11.924109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:11.924109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:11.924109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:11.928018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 351/702] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:11.929392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009b89bf-055f-449a-b61e-fe664aabbef0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:11.958780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:12.024078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7ebdc8-c3de-4062-90ee-5f40fe4ff9ce None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:12.045937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009b89bf-055f-449a-b61e-fe664aabbef0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:12.058649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa0593e-e9dd-424c-9cca-be20c14dcf06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:12.059243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa0593e-e9dd-424c-9cca-be20c14dcf06 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.059670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa0593e-e9dd-424c-9cca-be20c14dcf06 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.059882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa0593e-e9dd-424c-9cca-be20c14dcf06 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.060749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa0593e-e9dd-424c-9cca-be20c14dcf06 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:12.062197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa0593e-e9dd-424c-9cca-be20c14dcf06 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:12.076217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:12.082638 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3717e6-2c5b-4319-8bbc-1a94a89d54e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:12.083253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3717e6-2c5b-4319-8bbc-1a94a89d54e8 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.083531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3717e6-2c5b-4319-8bbc-1a94a89d54e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.083856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3717e6-2c5b-4319-8bbc-1a94a89d54e8 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.084961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3717e6-2c5b-4319-8bbc-1a94a89d54e8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:12.085274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3717e6-2c5b-4319-8bbc-1a94a89d54e8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:12.086724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3717e6-2c5b-4319-8bbc-1a94a89d54e8 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:12.094258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 352/703] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:42:11 2022] POST /identity/v3/users => generated 254 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:12.095967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa0593e-e9dd-424c-9cca-be20c14dcf06 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:12.100648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 352/704] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:12.101937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffac8f6-9bee-4fc8-86c9-b449448a8e83 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.102251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffac8f6-9bee-4fc8-86c9-b449448a8e83 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.102527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffac8f6-9bee-4fc8-86c9-b449448a8e83 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.103884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 353/705] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:12.108672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffac8f6-9bee-4fc8-86c9-b449448a8e83 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:12.246235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffac8f6-9bee-4fc8-86c9-b449448a8e83 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:12.257439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:12.270295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa0593e-e9dd-424c-9cca-be20c14dcf06 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:12.299067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fa89cc-634c-468a-a2cd-6932d2e8b6ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:12.299813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fa89cc-634c-468a-a2cd-6932d2e8b6ae None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.300187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fa89cc-634c-468a-a2cd-6932d2e8b6ae None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.300574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fa89cc-634c-468a-a2cd-6932d2e8b6ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.301178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fa89cc-634c-468a-a2cd-6932d2e8b6ae None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=dad93d99b865407aa120035015f4692a) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:12.301481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fa89cc-634c-468a-a2cd-6932d2e8b6ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:12.304675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fa89cc-634c-468a-a2cd-6932d2e8b6ae None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:12.310742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 354/706] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:12 2022] GET /identity/v3/role_assignments?user.id=dad93d99b865407aa120035015f4692a&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 259 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:12.311766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17530cf9-5528-468c-839e-cde27a876e4a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.312152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17530cf9-5528-468c-839e-cde27a876e4a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.312322 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17530cf9-5528-468c-839e-cde27a876e4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.327614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90229214-f23a-4a6f-b7d5-2684cabf2cf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:12.328293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90229214-f23a-4a6f-b7d5-2684cabf2cf9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.328624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90229214-f23a-4a6f-b7d5-2684cabf2cf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.328895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90229214-f23a-4a6f-b7d5-2684cabf2cf9 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.329553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90229214-f23a-4a6f-b7d5-2684cabf2cf9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:12.329910 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90229214-f23a-4a6f-b7d5-2684cabf2cf9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:12.333082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90229214-f23a-4a6f-b7d5-2684cabf2cf9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:12.339054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-17530cf9-5528-468c-839e-cde27a876e4a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:12.340272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-17530cf9-5528-468c-839e-cde27a876e4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:12.349992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 353/707] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:12.352875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaed5e3-c89b-4d1a-be7c-178d7ffaff44 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.353150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaed5e3-c89b-4d1a-be7c-178d7ffaff44 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.353408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaed5e3-c89b-4d1a-be7c-178d7ffaff44 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.354947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 354/708] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:12.355649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaed5e3-c89b-4d1a-be7c-178d7ffaff44 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:12.407666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:12.432938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 355/709] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:12.433134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f929a9-495a-4719-905e-f70fddec8c21 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.433449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f929a9-495a-4719-905e-f70fddec8c21 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.433717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f929a9-495a-4719-905e-f70fddec8c21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.453464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-32f929a9-495a-4719-905e-f70fddec8c21 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:12.454893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-32f929a9-495a-4719-905e-f70fddec8c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:12.536756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 356/710] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:12.564090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaed5e3-c89b-4d1a-be7c-178d7ffaff44 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:12.606226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccab433-e8e5-43a1-8004-226f5537dd80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:12.607154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccab433-e8e5-43a1-8004-226f5537dd80 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.607599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccab433-e8e5-43a1-8004-226f5537dd80 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.608062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccab433-e8e5-43a1-8004-226f5537dd80 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.617327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccab433-e8e5-43a1-8004-226f5537dd80 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:12.619077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccab433-e8e5-43a1-8004-226f5537dd80 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bccab433-e8e5-43a1-8004-226f5537dd80 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:12.630712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:12.631878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:12.633256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:12.633256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:12.633256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:12.633256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:12.633256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:12.633256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:12.633256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:12.633256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:12.635261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 355/711] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:12.642130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f929a9-495a-4719-905e-f70fddec8c21 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:12.736709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccab433-e8e5-43a1-8004-226f5537dd80 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:12.769132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f929a9-495a-4719-905e-f70fddec8c21 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:12.770498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:12.788453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:12.812993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cd5aec-9eac-425d-aa47-0573b3f04542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:12.813909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cd5aec-9eac-425d-aa47-0573b3f04542 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.815178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cd5aec-9eac-425d-aa47-0573b3f04542 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.815603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cd5aec-9eac-425d-aa47-0573b3f04542 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.816439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cd5aec-9eac-425d-aa47-0573b3f04542 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:12.816888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cd5aec-9eac-425d-aa47-0573b3f04542 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:12.818638 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cd5aec-9eac-425d-aa47-0573b3f04542 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:12.833024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 357/712] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:12.838659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cd5aec-9eac-425d-aa47-0573b3f04542 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:12.934284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccab433-e8e5-43a1-8004-226f5537dd80 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:12.978497 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cd5aec-9eac-425d-aa47-0573b3f04542 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:12.984628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f930b5-5e27-4c7a-bb0d-a1bc513fd142 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:12.985274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f930b5-5e27-4c7a-bb0d-a1bc513fd142 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:12.985643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f930b5-5e27-4c7a-bb0d-a1bc513fd142 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:12.986333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f930b5-5e27-4c7a-bb0d-a1bc513fd142 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:12.986922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f930b5-5e27-4c7a-bb0d-a1bc513fd142 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:12.988743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f930b5-5e27-4c7a-bb0d-a1bc513fd142 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-12f930b5-5e27-4c7a-bb0d-a1bc513fd142 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:13.005265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:13.007141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:13.008887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 356/713] 10.222.0.25 () {56 vars in 1129 bytes} [Sun Oct 2 00:42:12 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:13.009853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f930b5-5e27-4c7a-bb0d-a1bc513fd142 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:13.022425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8f5d7-cb1a-4bae-aa4b-9e4c65db5855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:13.022425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8f5d7-cb1a-4bae-aa4b-9e4c65db5855 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:13.022425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8f5d7-cb1a-4bae-aa4b-9e4c65db5855 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:13.022425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8f5d7-cb1a-4bae-aa4b-9e4c65db5855 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:13.023006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8f5d7-cb1a-4bae-aa4b-9e4c65db5855 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=db6c9dbbcf244a6f8bac59d2c9c973bc) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:13.023315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8f5d7-cb1a-4bae-aa4b-9e4c65db5855 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:13.024964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8f5d7-cb1a-4bae-aa4b-9e4c65db5855 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:13.038550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 358/714] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:12 2022] GET /identity/v3/role_assignments?user.id=db6c9dbbcf244a6f8bac59d2c9c973bc&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 608 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:13.039438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8f5d7-cb1a-4bae-aa4b-9e4c65db5855 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:13.149224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:13.167988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:13.199551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f930b5-5e27-4c7a-bb0d-a1bc513fd142 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:13.252029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987ead67-8b36-4cbc-9f62-1fc6e13c4543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:13.252883 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987ead67-8b36-4cbc-9f62-1fc6e13c4543 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:13.253171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987ead67-8b36-4cbc-9f62-1fc6e13c4543 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:13.253497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987ead67-8b36-4cbc-9f62-1fc6e13c4543 None admin] PATH_INFO: `/v3/services/network` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:13.254565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987ead67-8b36-4cbc-9f62-1fc6e13c4543 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:13.256523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987ead67-8b36-4cbc-9f62-1fc6e13c4543 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-987ead67-8b36-4cbc-9f62-1fc6e13c4543 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:13.269113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:13.270222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:13.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 357/715] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:13 2022] GET /identity/v3/services/network => generated 88 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:13.275947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8f5d7-cb1a-4bae-aa4b-9e4c65db5855 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:13.281202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987ead67-8b36-4cbc-9f62-1fc6e13c4543 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:13.317795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756cf6cf-3f93-45b2-a703-d5a88d1ff1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:13.318773 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756cf6cf-3f93-45b2-a703-d5a88d1ff1c7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:13.319333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756cf6cf-3f93-45b2-a703-d5a88d1ff1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:13.319935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756cf6cf-3f93-45b2-a703-d5a88d1ff1c7 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:13.320801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756cf6cf-3f93-45b2-a703-d5a88d1ff1c7 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:13.321474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756cf6cf-3f93-45b2-a703-d5a88d1ff1c7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:13.323978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756cf6cf-3f93-45b2-a703-d5a88d1ff1c7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:13.330853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 359/716] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 2 00:42:13 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:13.336692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756cf6cf-3f93-45b2-a703-d5a88d1ff1c7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:13.398948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987ead67-8b36-4cbc-9f62-1fc6e13c4543 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:13.438865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5a1538-83fa-4d13-ba18-8a89c854a348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:13.439469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5a1538-83fa-4d13-ba18-8a89c854a348 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:13.439743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5a1538-83fa-4d13-ba18-8a89c854a348 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:13.440010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5a1538-83fa-4d13-ba18-8a89c854a348 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:13.440627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5a1538-83fa-4d13-ba18-8a89c854a348 None admin] RBAC: Adding query filter params (name=network) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:13.440924 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5a1538-83fa-4d13-ba18-8a89c854a348 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:13.442309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5a1538-83fa-4d13-ba18-8a89c854a348 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:13.447838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 358/717] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:13 2022] GET /identity/v3/services?name=network => generated 124 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:13.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5a1538-83fa-4d13-ba18-8a89c854a348 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:13.461482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756cf6cf-3f93-45b2-a703-d5a88d1ff1c7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:13.463546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:13.465052 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:13.515203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affdb375-2d28-4915-bc3c-da464fb70679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:13.516386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affdb375-2d28-4915-bc3c-da464fb70679 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:13.516901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affdb375-2d28-4915-bc3c-da464fb70679 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:13.517566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affdb375-2d28-4915-bc3c-da464fb70679 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:13.518623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affdb375-2d28-4915-bc3c-da464fb70679 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:13.519150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affdb375-2d28-4915-bc3c-da464fb70679 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:13.520982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affdb375-2d28-4915-bc3c-da464fb70679 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:13.532606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 360/718] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 2 00:42:13 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:13.570760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5a1538-83fa-4d13-ba18-8a89c854a348 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:13.613839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ea98ed-2c72-4068-bb70-2fad917c9a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:13.614962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea98ed-2c72-4068-bb70-2fad917c9a72 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:13.615433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea98ed-2c72-4068-bb70-2fad917c9a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:13.616501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea98ed-2c72-4068-bb70-2fad917c9a72 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:13.617360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea98ed-2c72-4068-bb70-2fad917c9a72 None admin] RBAC: Adding query filter params (type=network) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:13.617878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea98ed-2c72-4068-bb70-2fad917c9a72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:13.619563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea98ed-2c72-4068-bb70-2fad917c9a72 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:13.627023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 359/719] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:13 2022] GET /identity/v3/services?type=network => generated 124 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:14.685426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e87c72-6ce2-47a7-add5-c7cc1cfea55b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:14.686468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e87c72-6ce2-47a7-add5-c7cc1cfea55b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:14.686981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e87c72-6ce2-47a7-add5-c7cc1cfea55b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:14.688833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 361/720] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:14.693806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf5a5ad-c4c7-4447-9682-7cc4ea95be95 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:14.694606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf5a5ad-c4c7-4447-9682-7cc4ea95be95 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:14.695104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf5a5ad-c4c7-4447-9682-7cc4ea95be95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:14.718447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-abf5a5ad-c4c7-4447-9682-7cc4ea95be95 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:14.720004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-abf5a5ad-c4c7-4447-9682-7cc4ea95be95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:14.807624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:14.818266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 360/721] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:14.960732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb9fd8-658d-4e29-b3f2-b58742812b15 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:14.960919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb9fd8-658d-4e29-b3f2-b58742812b15 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:14.961188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cb9fd8-658d-4e29-b3f2-b58742812b15 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:14.962531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 362/722] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:14.971128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130ec51e-42f0-49fc-924e-ef807deb3e67 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:14.971619 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130ec51e-42f0-49fc-924e-ef807deb3e67 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:14.972130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130ec51e-42f0-49fc-924e-ef807deb3e67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:14.995392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-130ec51e-42f0-49fc-924e-ef807deb3e67 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:14.996938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-130ec51e-42f0-49fc-924e-ef807deb3e67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:15.091392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 361/723] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:15.239220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb9fd8-658d-4e29-b3f2-b58742812b15 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:15.342515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cb9fd8-658d-4e29-b3f2-b58742812b15 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:15.379315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b25c2b-fc13-497c-946c-540f620a87af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:15.379939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b25c2b-fc13-497c-946c-540f620a87af None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:15.380216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b25c2b-fc13-497c-946c-540f620a87af None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:15.380484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b25c2b-fc13-497c-946c-540f620a87af None admin] PATH_INFO: `/v3/roles/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:15.390854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b25c2b-fc13-497c-946c-540f620a87af None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-37b25c2b-fc13-497c-946c-540f620a87af None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:15.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:15.397952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:15.398829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 363/724] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:15.405170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130ec51e-42f0-49fc-924e-ef807deb3e67 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:15.451418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b25c2b-fc13-497c-946c-540f620a87af None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:15.500311 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:15.545875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130ec51e-42f0-49fc-924e-ef807deb3e67 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:15.603614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:15.611308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf64553-d39d-4c15-a266-bf7b7540e186 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:15.612396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf64553-d39d-4c15-a266-bf7b7540e186 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:15.612673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf64553-d39d-4c15-a266-bf7b7540e186 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:15.612944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf64553-d39d-4c15-a266-bf7b7540e186 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:15.613605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf64553-d39d-4c15-a266-bf7b7540e186 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:15.613904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf64553-d39d-4c15-a266-bf7b7540e186 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:15.616222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf64553-d39d-4c15-a266-bf7b7540e186 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:15.625946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 362/725] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:42:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:15.632294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf64553-d39d-4c15-a266-bf7b7540e186 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:15.658276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b25c2b-fc13-497c-946c-540f620a87af None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:15.697133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcf6437-2372-4946-96f7-26c3dc901449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:15.697807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcf6437-2372-4946-96f7-26c3dc901449 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:15.698043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcf6437-2372-4946-96f7-26c3dc901449 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:15.698331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcf6437-2372-4946-96f7-26c3dc901449 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:15.710013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebcf6437-2372-4946-96f7-26c3dc901449 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-ebcf6437-2372-4946-96f7-26c3dc901449 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:15.715456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:15.716398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:15.717234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 364/726] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:15.728506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcf6437-2372-4946-96f7-26c3dc901449 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:15.822503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf64553-d39d-4c15-a266-bf7b7540e186 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:15.882526 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de8290b-c57c-479d-8395-8d87b22333a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:15.884455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de8290b-c57c-479d-8395-8d87b22333a8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:15.884868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:15.886103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de8290b-c57c-479d-8395-8d87b22333a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:15.889238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de8290b-c57c-479d-8395-8d87b22333a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:15.903668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de8290b-c57c-479d-8395-8d87b22333a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:15.906154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de8290b-c57c-479d-8395-8d87b22333a8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:15.921392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcf6437-2372-4946-96f7-26c3dc901449 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-3de8290b-c57c-479d-8395-8d87b22333a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:15.936666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:15.937829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:15.938950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:15.938950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:15.938950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:15.942866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 363/727] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:15.949172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de8290b-c57c-479d-8395-8d87b22333a8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:15.990208 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dfb3bf-fb7d-4b9b-987e-1ee0856ad81c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:15.991273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dfb3bf-fb7d-4b9b-987e-1ee0856ad81c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:15.991814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dfb3bf-fb7d-4b9b-987e-1ee0856ad81c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:15.992308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dfb3bf-fb7d-4b9b-987e-1ee0856ad81c None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:15.993275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dfb3bf-fb7d-4b9b-987e-1ee0856ad81c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:15.993817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dfb3bf-fb7d-4b9b-987e-1ee0856ad81c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:15.995457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dfb3bf-fb7d-4b9b-987e-1ee0856ad81c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:16.005001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 365/728] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:42:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:16.011705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dfb3bf-fb7d-4b9b-987e-1ee0856ad81c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:16.036050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:16.102515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de8290b-c57c-479d-8395-8d87b22333a8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:16.139945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dfb3bf-fb7d-4b9b-987e-1ee0856ad81c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:16.146469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bae0882-38c1-417c-8a2c-515b398211c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:16.147633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bae0882-38c1-417c-8a2c-515b398211c2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.147909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bae0882-38c1-417c-8a2c-515b398211c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.148176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bae0882-38c1-417c-8a2c-515b398211c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.149590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bae0882-38c1-417c-8a2c-515b398211c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:16.149918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bae0882-38c1-417c-8a2c-515b398211c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:16.151919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bae0882-38c1-417c-8a2c-515b398211c2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:16.169538 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 364/729] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:16.170928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cf5ee4-2dc7-4544-bcca-3eb405e97ac1 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.171205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cf5ee4-2dc7-4544-bcca-3eb405e97ac1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.171476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cf5ee4-2dc7-4544-bcca-3eb405e97ac1 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.173053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 365/730] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:16.174316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cf5ee4-2dc7-4544-bcca-3eb405e97ac1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:16.188846 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a518df36-c7c2-4c5e-b30d-7d7dcb5bb056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:16.189568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a518df36-c7c2-4c5e-b30d-7d7dcb5bb056 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.189848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a518df36-c7c2-4c5e-b30d-7d7dcb5bb056 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.190144 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a518df36-c7c2-4c5e-b30d-7d7dcb5bb056 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.202607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a518df36-c7c2-4c5e-b30d-7d7dcb5bb056 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:16.204321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a518df36-c7c2-4c5e-b30d-7d7dcb5bb056 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:16.214030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-a518df36-c7c2-4c5e-b30d-7d7dcb5bb056 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:16.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:16.222714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:16.224664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:16.224664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:16.224664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:16.224664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 366/731] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:16.225267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ceeaa7-1a9f-4f7d-94b9-818d9fa98900 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.225267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ceeaa7-1a9f-4f7d-94b9-818d9fa98900 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.225564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ceeaa7-1a9f-4f7d-94b9-818d9fa98900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.248174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-29ceeaa7-1a9f-4f7d-94b9-818d9fa98900 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:16.249358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-29ceeaa7-1a9f-4f7d-94b9-818d9fa98900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:16.294450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cf5ee4-2dc7-4544-bcca-3eb405e97ac1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:16.298776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:16.331676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 367/732] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:16.331676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ceeaa7-1a9f-4f7d-94b9-818d9fa98900 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:16.339602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a109f8-b242-419a-909a-746cb0026ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:16.340236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a109f8-b242-419a-909a-746cb0026ee0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.340512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a109f8-b242-419a-909a-746cb0026ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.340781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a109f8-b242-419a-909a-746cb0026ee0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.349114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a109f8-b242-419a-909a-746cb0026ee0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:16.351355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a109f8-b242-419a-909a-746cb0026ee0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-25a109f8-b242-419a-909a-746cb0026ee0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:16.364108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:16.365035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:16.365997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 366/733] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 2 00:42:16 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:16.367108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0f9ce-ec18-40fd-a215-789646c70d4f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.367204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0f9ce-ec18-40fd-a215-789646c70d4f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.371822 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0f9ce-ec18-40fd-a215-789646c70d4f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.372980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 367/734] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:16 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:16.373785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0f9ce-ec18-40fd-a215-789646c70d4f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:16.445578 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ceeaa7-1a9f-4f7d-94b9-818d9fa98900 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:16.474492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:16.480928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdc37e0-b0f7-43bc-8d09-47bd0eed978d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:16.481528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdc37e0-b0f7-43bc-8d09-47bd0eed978d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.481969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdc37e0-b0f7-43bc-8d09-47bd0eed978d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.482281 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdc37e0-b0f7-43bc-8d09-47bd0eed978d None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.483352 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdc37e0-b0f7-43bc-8d09-47bd0eed978d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:16.483600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdc37e0-b0f7-43bc-8d09-47bd0eed978d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:16.484901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdc37e0-b0f7-43bc-8d09-47bd0eed978d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:16.489283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0f9ce-ec18-40fd-a215-789646c70d4f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:16.496003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 368/735] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:16.499403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6aab0d-07f8-46dc-a35b-04454c4b7a88 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.500352 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6aab0d-07f8-46dc-a35b-04454c4b7a88 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.500634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6aab0d-07f8-46dc-a35b-04454c4b7a88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.521944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8c6aab0d-07f8-46dc-a35b-04454c4b7a88 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:16.523327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8c6aab0d-07f8-46dc-a35b-04454c4b7a88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:16.525823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b428d38-7f67-40b1-9ffa-f5b6aa745a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:16.527244 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b428d38-7f67-40b1-9ffa-f5b6aa745a90 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.527519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b428d38-7f67-40b1-9ffa-f5b6aa745a90 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.527786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b428d38-7f67-40b1-9ffa-f5b6aa745a90 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.528548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b428d38-7f67-40b1-9ffa-f5b6aa745a90 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:16.529066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b428d38-7f67-40b1-9ffa-f5b6aa745a90 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:16.530964 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b428d38-7f67-40b1-9ffa-f5b6aa745a90 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:16.597633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 369/736] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:16.597633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3733702c-0cd3-40d1-8c1b-04cffc94ae43 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.597633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3733702c-0cd3-40d1-8c1b-04cffc94ae43 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.597633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3733702c-0cd3-40d1-8c1b-04cffc94ae43 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.598435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 370/737] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:16 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:16.599689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3733702c-0cd3-40d1-8c1b-04cffc94ae43 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:16.675953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 368/738] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Oct 2 00:42:16 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:16.677099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdde1291-1798-4903-babd-626dd4202a60 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.677376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdde1291-1798-4903-babd-626dd4202a60 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.677641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdde1291-1798-4903-babd-626dd4202a60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.698008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bdde1291-1798-4903-babd-626dd4202a60 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:16.699211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bdde1291-1798-4903-babd-626dd4202a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:16.716605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:16.722119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3733702c-0cd3-40d1-8c1b-04cffc94ae43 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:16.743204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:16.757869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa9b79f-d5c8-4355-a174-88bcd5866151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:16.758515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa9b79f-d5c8-4355-a174-88bcd5866151 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.758788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa9b79f-d5c8-4355-a174-88bcd5866151 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.759054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa9b79f-d5c8-4355-a174-88bcd5866151 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.766684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa9b79f-d5c8-4355-a174-88bcd5866151 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:16.768295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa9b79f-d5c8-4355-a174-88bcd5866151 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:16.781939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 369/739] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 02 00:42:16.781939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdde1291-1798-4903-babd-626dd4202a60 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-bfa9b79f-d5c8-4355-a174-88bcd5866151 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:16.782189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:16.783053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 371/740] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 2 00:42:16 2022] GET /identity/v3/users/nova => generated 82 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:16.784010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa9b79f-d5c8-4355-a174-88bcd5866151 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:16.898431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa9b79f-d5c8-4355-a174-88bcd5866151 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:16.904089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdde1291-1798-4903-babd-626dd4202a60 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:16.933121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdce291-f9a9-45ec-aff1-ef3dc7d30f6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:16.933656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdce291-f9a9-45ec-aff1-ef3dc7d30f6d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.933932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdce291-f9a9-45ec-aff1-ef3dc7d30f6d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.934229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdce291-f9a9-45ec-aff1-ef3dc7d30f6d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.946204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3aa8a1-8745-4996-80d8-05ca907da0bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:16.946839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3aa8a1-8745-4996-80d8-05ca907da0bd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:16.947105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3aa8a1-8745-4996-80d8-05ca907da0bd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:16.947368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3aa8a1-8745-4996-80d8-05ca907da0bd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:16.958807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3aa8a1-8745-4996-80d8-05ca907da0bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4a3aa8a1-8745-4996-80d8-05ca907da0bd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:16.964050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:16.964993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:16.965839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 370/741] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:16.967048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3aa8a1-8745-4996-80d8-05ca907da0bd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:16.982835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:17.002029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:17.032916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdce291-f9a9-45ec-aff1-ef3dc7d30f6d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:17.034342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fdce291-f9a9-45ec-aff1-ef3dc7d30f6d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:17.080515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3aa8a1-8745-4996-80d8-05ca907da0bd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:17.112554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a9451b-235f-41f7-b1f5-72810586cea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:17.113149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a9451b-235f-41f7-b1f5-72810586cea1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:17.113427 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a9451b-235f-41f7-b1f5-72810586cea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:17.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a9451b-235f-41f7-b1f5-72810586cea1 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:17.114449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a9451b-235f-41f7-b1f5-72810586cea1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:17.114750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a9451b-235f-41f7-b1f5-72810586cea1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:17.116708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a9451b-235f-41f7-b1f5-72810586cea1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:17.133007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 372/742] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 02 00:42:17.133877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 371/743] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 2 00:42:16 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:17.133989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdce291-f9a9-45ec-aff1-ef3dc7d30f6d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:17.134529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a9451b-235f-41f7-b1f5-72810586cea1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:17.198670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:17.210429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:17.229305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a9451b-235f-41f7-b1f5-72810586cea1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:17.229695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdce291-f9a9-45ec-aff1-ef3dc7d30f6d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:17.262426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b82cbe4-cafb-4929-b7a6-bf8eb7b472d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:17.263011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b82cbe4-cafb-4929-b7a6-bf8eb7b472d8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:17.263284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b82cbe4-cafb-4929-b7a6-bf8eb7b472d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:17.263545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b82cbe4-cafb-4929-b7a6-bf8eb7b472d8 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:17.264182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b82cbe4-cafb-4929-b7a6-bf8eb7b472d8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:17.264284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a649e5-6d61-4907-895f-4c016c2670a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:17.264472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b82cbe4-cafb-4929-b7a6-bf8eb7b472d8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:17.264868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a649e5-6d61-4907-895f-4c016c2670a4 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:17.265148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a649e5-6d61-4907-895f-4c016c2670a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:17.265419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a649e5-6d61-4907-895f-4c016c2670a4 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:17.265808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b82cbe4-cafb-4929-b7a6-bf8eb7b472d8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:17.266074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a649e5-6d61-4907-895f-4c016c2670a4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:17.267435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a649e5-6d61-4907-895f-4c016c2670a4 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:17.273710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 372/744] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:42:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:17.274413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b82cbe4-cafb-4929-b7a6-bf8eb7b472d8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:17.290157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 373/745] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:42:17 2022] POST /identity/v3/services => generated 240 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:17.290157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a649e5-6d61-4907-895f-4c016c2670a4 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:17.412783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b82cbe4-cafb-4929-b7a6-bf8eb7b472d8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:17.421146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a649e5-6d61-4907-895f-4c016c2670a4 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:17.445762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:17.451306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f85af7-ddbe-4783-a057-aa9a7814fa74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:17.451851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f85af7-ddbe-4783-a057-aa9a7814fa74 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:17.452127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f85af7-ddbe-4783-a057-aa9a7814fa74 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:17.452402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f85af7-ddbe-4783-a057-aa9a7814fa74 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:17.453051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f85af7-ddbe-4783-a057-aa9a7814fa74 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:17.454447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f85af7-ddbe-4783-a057-aa9a7814fa74 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:17.464877 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20be083-a05f-44c6-8a8f-10d94bba1ce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:17.466105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20be083-a05f-44c6-8a8f-10d94bba1ce4 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:17.466105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20be083-a05f-44c6-8a8f-10d94bba1ce4 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:17.466309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20be083-a05f-44c6-8a8f-10d94bba1ce4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:17.469731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:17.478099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 373/746] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:42:17 2022] POST /identity/v3/services => generated 251 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:17.478099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f85af7-ddbe-4783-a057-aa9a7814fa74 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:17.479078 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20be083-a05f-44c6-8a8f-10d94bba1ce4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:17.480563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20be083-a05f-44c6-8a8f-10d94bba1ce4 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-a20be083-a05f-44c6-8a8f-10d94bba1ce4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:17.499463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:17.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:17.501467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:17.501467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:17.501467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:17.501467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 374/747] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:17.501467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20be083-a05f-44c6-8a8f-10d94bba1ce4 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:17.628958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f85af7-ddbe-4783-a057-aa9a7814fa74 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:17.638205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20be083-a05f-44c6-8a8f-10d94bba1ce4 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:17.671319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af89bf0-c62c-4f66-85c2-7839253069df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:17.671946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af89bf0-c62c-4f66-85c2-7839253069df None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:17.672218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af89bf0-c62c-4f66-85c2-7839253069df None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:17.672485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af89bf0-c62c-4f66-85c2-7839253069df None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:17.683976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af89bf0-c62c-4f66-85c2-7839253069df None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:17.685426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af89bf0-c62c-4f66-85c2-7839253069df None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:17.687192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2387e77c-ad79-487c-a0bd-a2c79a4ae067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:17.687813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2387e77c-ad79-487c-a0bd-a2c79a4ae067 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:17.688131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2387e77c-ad79-487c-a0bd-a2c79a4ae067 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:17.688350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2387e77c-ad79-487c-a0bd-a2c79a4ae067 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:17.701986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2387e77c-ad79-487c-a0bd-a2c79a4ae067 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1af89bf0-c62c-4f66-85c2-7839253069df None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:17.703324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:17.704207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:17.705096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:17.705096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:17.705096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:17.705265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2387e77c-ad79-487c-a0bd-a2c79a4ae067 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:17.705335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 374/748] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:17.705335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af89bf0-c62c-4f66-85c2-7839253069df None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-2387e77c-ad79-487c-a0bd-a2c79a4ae067 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:17.721725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:17.723330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:17.724215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:17.724215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:17.724215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:17.724215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 375/749] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:17.724215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2387e77c-ad79-487c-a0bd-a2c79a4ae067 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:17.770504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:17.772597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:17.827467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af89bf0-c62c-4f66-85c2-7839253069df None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:17.838689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2387e77c-ad79-487c-a0bd-a2c79a4ae067 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:17.864612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99af88a1-7a93-4642-93e0-d2072c7fdb7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:17.866050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99af88a1-7a93-4642-93e0-d2072c7fdb7e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:17.866653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99af88a1-7a93-4642-93e0-d2072c7fdb7e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:17.867058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99af88a1-7a93-4642-93e0-d2072c7fdb7e None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:17.867926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99af88a1-7a93-4642-93e0-d2072c7fdb7e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:17.868363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99af88a1-7a93-4642-93e0-d2072c7fdb7e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:17.869953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99af88a1-7a93-4642-93e0-d2072c7fdb7e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:17.875732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff600158-606f-40ff-ab3e-594347050aa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:17.876459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff600158-606f-40ff-ab3e-594347050aa1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:17.876825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff600158-606f-40ff-ab3e-594347050aa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:17.877186 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff600158-606f-40ff-ab3e-594347050aa1 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:17.878220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff600158-606f-40ff-ab3e-594347050aa1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:17.878464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff600158-606f-40ff-ab3e-594347050aa1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:17.880381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff600158-606f-40ff-ab3e-594347050aa1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:17.888346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 375/750] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:17.889514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99af88a1-7a93-4642-93e0-d2072c7fdb7e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:17.893876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 376/751] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:17.894240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff600158-606f-40ff-ab3e-594347050aa1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:17.994317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:17.997713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:17.998906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99af88a1-7a93-4642-93e0-d2072c7fdb7e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:18.002093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff600158-606f-40ff-ab3e-594347050aa1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:18.042204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c031ae-c38d-4b36-b019-27fa35de7050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:18.043128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c031ae-c38d-4b36-b019-27fa35de7050 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:18.043552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c031ae-c38d-4b36-b019-27fa35de7050 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:18.044028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c031ae-c38d-4b36-b019-27fa35de7050 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:18.046537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d80839-dbc3-4524-8f56-ba5efe3d6650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:18.047388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d80839-dbc3-4524-8f56-ba5efe3d6650 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:18.047718 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d80839-dbc3-4524-8f56-ba5efe3d6650 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:18.048079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d80839-dbc3-4524-8f56-ba5efe3d6650 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:18.048810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d80839-dbc3-4524-8f56-ba5efe3d6650 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:18.049199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d80839-dbc3-4524-8f56-ba5efe3d6650 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:18.050668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d80839-dbc3-4524-8f56-ba5efe3d6650 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:18.052646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c031ae-c38d-4b36-b019-27fa35de7050 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:18.054502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c031ae-c38d-4b36-b019-27fa35de7050 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:18.063635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 376/752] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:18.064644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d80839-dbc3-4524-8f56-ba5efe3d6650 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-26c031ae-c38d-4b36-b019-27fa35de7050 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:18.068395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:18.069792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:18.070735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:18.070735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:18.070735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:18.070735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:18.070735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:18.070735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:18.070735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:18.070735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:18.073089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 377/753] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:18.074071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c031ae-c38d-4b36-b019-27fa35de7050 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:18.175216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d80839-dbc3-4524-8f56-ba5efe3d6650 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:18.179245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c031ae-c38d-4b36-b019-27fa35de7050 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:18.211674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae76716-9d3f-4431-ace4-d509ea21259b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:18.212401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae76716-9d3f-4431-ace4-d509ea21259b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:18.212721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae76716-9d3f-4431-ace4-d509ea21259b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:18.213055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae76716-9d3f-4431-ace4-d509ea21259b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:18.216589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f390156-ee65-4596-8575-442074b48e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:18.217465 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f390156-ee65-4596-8575-442074b48e69 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:18.217966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f390156-ee65-4596-8575-442074b48e69 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:18.218423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f390156-ee65-4596-8575-442074b48e69 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:18.220587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae76716-9d3f-4431-ace4-d509ea21259b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:18.222250 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae76716-9d3f-4431-ace4-d509ea21259b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:18.227929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f390156-ee65-4596-8575-442074b48e69 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:18.229720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f390156-ee65-4596-8575-442074b48e69 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-eae76716-9d3f-4431-ace4-d509ea21259b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:18.235574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:18.236557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:18.237454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:18.237454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:18.237454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:18.237454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:18.237454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:18.237454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:18.237454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:18.237454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:18.237454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 377/754] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:18.238045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae76716-9d3f-4431-ace4-d509ea21259b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-3f390156-ee65-4596-8575-442074b48e69 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:18.246409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:18.247457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:18.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:18.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:18.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:18.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:18.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:18.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:18.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:18.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:18.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:42:18.248397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:18.250661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 378/755] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:18.251726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f390156-ee65-4596-8575-442074b48e69 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:18.264375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:18.279406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:18.348230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae76716-9d3f-4431-ace4-d509ea21259b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:18.364833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f390156-ee65-4596-8575-442074b48e69 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:18.384537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740591b8-19c4-4d76-9247-4c3d11964e32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:18.385314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740591b8-19c4-4d76-9247-4c3d11964e32 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:18.385683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740591b8-19c4-4d76-9247-4c3d11964e32 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:18.386044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740591b8-19c4-4d76-9247-4c3d11964e32 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:18.386839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740591b8-19c4-4d76-9247-4c3d11964e32 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:18.387633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740591b8-19c4-4d76-9247-4c3d11964e32 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:18.388856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740591b8-19c4-4d76-9247-4c3d11964e32 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:18.402364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d07e66d-bf50-472c-9041-a131cbc68321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:18.403253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d07e66d-bf50-472c-9041-a131cbc68321 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:18.403744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d07e66d-bf50-472c-9041-a131cbc68321 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:18.404204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d07e66d-bf50-472c-9041-a131cbc68321 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:18.405075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d07e66d-bf50-472c-9041-a131cbc68321 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:18.405605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d07e66d-bf50-472c-9041-a131cbc68321 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:18.406799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 378/756] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:18.407660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740591b8-19c4-4d76-9247-4c3d11964e32 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:18.407913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d07e66d-bf50-472c-9041-a131cbc68321 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:18.422137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 379/757] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:18.423531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d07e66d-bf50-472c-9041-a131cbc68321 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:18.494884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:18.511822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:18.516441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740591b8-19c4-4d76-9247-4c3d11964e32 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:18.533137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d07e66d-bf50-472c-9041-a131cbc68321 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:18.554972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146a4125-6523-434e-a66a-5325a0156c41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:18.555357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a4125-6523-434e-a66a-5325a0156c41 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:18.555633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a4125-6523-434e-a66a-5325a0156c41 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:18.555901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a4125-6523-434e-a66a-5325a0156c41 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:18.556620 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146a4125-6523-434e-a66a-5325a0156c41 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:18.556963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146a4125-6523-434e-a66a-5325a0156c41 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:18.558487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146a4125-6523-434e-a66a-5325a0156c41 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:18.570523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa14a7b9-cf81-4088-b3fa-fedd039290ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:18.571414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa14a7b9-cf81-4088-b3fa-fedd039290ae None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:18.571886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa14a7b9-cf81-4088-b3fa-fedd039290ae None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:18.572360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa14a7b9-cf81-4088-b3fa-fedd039290ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:18.573393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa14a7b9-cf81-4088-b3fa-fedd039290ae None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=af51bf378ea2491780250fc037fa2920) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:18.573894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa14a7b9-cf81-4088-b3fa-fedd039290ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:18.575525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa14a7b9-cf81-4088-b3fa-fedd039290ae None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:18.576883 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 379/758] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 2 00:42:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:18.577779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146a4125-6523-434e-a66a-5325a0156c41 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:18.586789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 380/759] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:18 2022] GET /identity/v3/role_assignments?user.id=af51bf378ea2491780250fc037fa2920&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 259 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:18.588012 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa14a7b9-cf81-4088-b3fa-fedd039290ae None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:18.715633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146a4125-6523-434e-a66a-5325a0156c41 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:18.781654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa14a7b9-cf81-4088-b3fa-fedd039290ae None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:18.784386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e600f0-ff88-4b2b-8203-fe5532f05781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:18.785067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e600f0-ff88-4b2b-8203-fe5532f05781 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:18.785338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e600f0-ff88-4b2b-8203-fe5532f05781 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:18.785603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e600f0-ff88-4b2b-8203-fe5532f05781 None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/dad93d99b865407aa120035015f4692a/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:18.788109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e600f0-ff88-4b2b-8203-fe5532f05781 None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=dad93d99b865407aa120035015f4692a, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:18.789571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e600f0-ff88-4b2b-8203-fe5532f05781 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:18.809248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:18.830908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 380/760] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:42:18 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/dad93d99b865407aa120035015f4692a/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:42:18.831836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c59ead-b120-4560-9830-f0cbe3a03147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:18.832605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c59ead-b120-4560-9830-f0cbe3a03147 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:18.833084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c59ead-b120-4560-9830-f0cbe3a03147 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:18.833532 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c59ead-b120-4560-9830-f0cbe3a03147 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:18.861556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:18.941543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c59ead-b120-4560-9830-f0cbe3a03147 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:18.943353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c59ead-b120-4560-9830-f0cbe3a03147 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:19.081569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 381/761] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:18 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:42:19.087380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e600f0-ff88-4b2b-8203-fe5532f05781 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:19.197195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e600f0-ff88-4b2b-8203-fe5532f05781 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:19.231437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89a8c7d-a109-4706-afa8-eeba7bf8889e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:19.232062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a8c7d-a109-4706-afa8-eeba7bf8889e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:19.232332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a8c7d-a109-4706-afa8-eeba7bf8889e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:19.232605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a8c7d-a109-4706-afa8-eeba7bf8889e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:19.244005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89a8c7d-a109-4706-afa8-eeba7bf8889e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:19.245449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89a8c7d-a109-4706-afa8-eeba7bf8889e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c89a8c7d-a109-4706-afa8-eeba7bf8889e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:19.260241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:19.261456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:19.262774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:19.262774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:19.262774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:19.262774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 381/762] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:19.266855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c59ead-b120-4560-9830-f0cbe3a03147 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:19.338854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:19.413224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c59ead-b120-4560-9830-f0cbe3a03147 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:19.448954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abade59c-cbfa-463d-98c0-e0c8422b0af9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:19.449828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abade59c-cbfa-463d-98c0-e0c8422b0af9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:19.450317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abade59c-cbfa-463d-98c0-e0c8422b0af9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:19.450737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abade59c-cbfa-463d-98c0-e0c8422b0af9 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:19.451537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abade59c-cbfa-463d-98c0-e0c8422b0af9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:19.452007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abade59c-cbfa-463d-98c0-e0c8422b0af9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:19.453528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abade59c-cbfa-463d-98c0-e0c8422b0af9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:19.466307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 382/763] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:19.471246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89a8c7d-a109-4706-afa8-eeba7bf8889e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:19.529466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:19.589918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89a8c7d-a109-4706-afa8-eeba7bf8889e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:19.625144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eeae4a3-feaa-449c-99e0-b98433bf8289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:19.625774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eeae4a3-feaa-449c-99e0-b98433bf8289 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:19.626052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eeae4a3-feaa-449c-99e0-b98433bf8289 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:19.626417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eeae4a3-feaa-449c-99e0-b98433bf8289 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:19.634154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eeae4a3-feaa-449c-99e0-b98433bf8289 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:19.635594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eeae4a3-feaa-449c-99e0-b98433bf8289 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-9eeae4a3-feaa-449c-99e0-b98433bf8289 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:19.654324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:19.655314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:19.656230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:19.656230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:19.656230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:19.656230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:19.656230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:19.656230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:19.656230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:19.656230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:19.656230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 382/764] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:19.663787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abade59c-cbfa-463d-98c0-e0c8422b0af9 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:19.807844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abade59c-cbfa-463d-98c0-e0c8422b0af9 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:19.815097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:19.863198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00885635-e496-4b75-aa0a-5185f64bc97d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:19.863726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00885635-e496-4b75-aa0a-5185f64bc97d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:19.863995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00885635-e496-4b75-aa0a-5185f64bc97d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:19.864265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00885635-e496-4b75-aa0a-5185f64bc97d None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:19.864953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00885635-e496-4b75-aa0a-5185f64bc97d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:19.865823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00885635-e496-4b75-aa0a-5185f64bc97d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:19.867453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00885635-e496-4b75-aa0a-5185f64bc97d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:19.881431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dbd033-82f7-4f52-8f23-7e45289822c5 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:19.881740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dbd033-82f7-4f52-8f23-7e45289822c5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:19.882021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dbd033-82f7-4f52-8f23-7e45289822c5 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:19.883949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 383/765] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:19.885785 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 383/766] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:19.891109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96e81d9-ccfb-48bc-8456-5dc1dab32f51 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:19.891368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96e81d9-ccfb-48bc-8456-5dc1dab32f51 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:19.891620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00885635-e496-4b75-aa0a-5185f64bc97d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:19.891723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96e81d9-ccfb-48bc-8456-5dc1dab32f51 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:19.893346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 384/767] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:19.894390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602c260e-a918-41b1-87ec-bb00e972998f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:19.894665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602c260e-a918-41b1-87ec-bb00e972998f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:19.894932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602c260e-a918-41b1-87ec-bb00e972998f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:19.918680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-602c260e-a918-41b1-87ec-bb00e972998f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:19.921289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-602c260e-a918-41b1-87ec-bb00e972998f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:19.993559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 385/768] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:19 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:19.995736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fb363a-85e6-4db2-82e7-2ecc60a01c48 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:19.996273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fb363a-85e6-4db2-82e7-2ecc60a01c48 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:19.996864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fb363a-85e6-4db2-82e7-2ecc60a01c48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:20.000099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00885635-e496-4b75-aa0a-5185f64bc97d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:20.018684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e8fb363a-85e6-4db2-82e7-2ecc60a01c48 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:20.020413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e8fb363a-85e6-4db2-82e7-2ecc60a01c48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:20.044211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce67ec86-536a-47bb-80ba-50a5b45d2b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:20.044728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67ec86-536a-47bb-80ba-50a5b45d2b5a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:20.045012 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67ec86-536a-47bb-80ba-50a5b45d2b5a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:20.045284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67ec86-536a-47bb-80ba-50a5b45d2b5a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:20.046107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce67ec86-536a-47bb-80ba-50a5b45d2b5a None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=db6c9dbbcf244a6f8bac59d2c9c973bc) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:20.046366 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce67ec86-536a-47bb-80ba-50a5b45d2b5a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:20.047897 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce67ec86-536a-47bb-80ba-50a5b45d2b5a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:20.062285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 384/769] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:19 2022] GET /identity/v3/role_assignments?user.id=db6c9dbbcf244a6f8bac59d2c9c973bc&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:20.077482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:20.114002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 386/770] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:19 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:20.325548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce67ec86-536a-47bb-80ba-50a5b45d2b5a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:20.356008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:20.430017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce67ec86-536a-47bb-80ba-50a5b45d2b5a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:20.464250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68128009-18aa-4822-bd25-6d2c83dc0db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:20.464936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68128009-18aa-4822-bd25-6d2c83dc0db3 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:20.465222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68128009-18aa-4822-bd25-6d2c83dc0db3 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:20.465580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68128009-18aa-4822-bd25-6d2c83dc0db3 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:20.466186 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68128009-18aa-4822-bd25-6d2c83dc0db3 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:20.467662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68128009-18aa-4822-bd25-6d2c83dc0db3 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-68128009-18aa-4822-bd25-6d2c83dc0db3 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:20.479740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:20.480751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:20.482024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 385/771] 10.222.0.25 () {56 vars in 1129 bytes} [Sun Oct 2 00:42:20 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:20.487109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fb363a-85e6-4db2-82e7-2ecc60a01c48 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:20.518958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68128009-18aa-4822-bd25-6d2c83dc0db3 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:20.673053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fb363a-85e6-4db2-82e7-2ecc60a01c48 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:20.714304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e1eaa-631e-44bf-aef9-cb65145b9c7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:20.714827 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e1eaa-631e-44bf-aef9-cb65145b9c7e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:20.715096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e1eaa-631e-44bf-aef9-cb65145b9c7e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:20.715516 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e1eaa-631e-44bf-aef9-cb65145b9c7e None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:20.716141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e1eaa-631e-44bf-aef9-cb65145b9c7e None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:20.716437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e1eaa-631e-44bf-aef9-cb65145b9c7e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:20.717845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e1eaa-631e-44bf-aef9-cb65145b9c7e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:20.720063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:20.723455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 387/772] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 2 00:42:20 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:20.729280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e1eaa-631e-44bf-aef9-cb65145b9c7e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:20.740722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68128009-18aa-4822-bd25-6d2c83dc0db3 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:20.768780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:20.827346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28ce113-3725-4324-8266-056b9168f075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:20.828039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28ce113-3725-4324-8266-056b9168f075 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:20.828261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28ce113-3725-4324-8266-056b9168f075 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:20.828533 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28ce113-3725-4324-8266-056b9168f075 None admin] PATH_INFO: `/v3/services/network` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:20.829119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28ce113-3725-4324-8266-056b9168f075 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:20.830516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28ce113-3725-4324-8266-056b9168f075 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:20.843300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e1eaa-631e-44bf-aef9-cb65145b9c7e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-d28ce113-3725-4324-8266-056b9168f075 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:20.851600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:20.853873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 386/773] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:20 2022] GET /identity/v3/services/network => generated 88 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:20.863012 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28ce113-3725-4324-8266-056b9168f075 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:20.879438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184bd643-6a0d-4d12-8e10-dfb0957cd748 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:20.880044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184bd643-6a0d-4d12-8e10-dfb0957cd748 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:20.880358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184bd643-6a0d-4d12-8e10-dfb0957cd748 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:20.880627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184bd643-6a0d-4d12-8e10-dfb0957cd748 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:20.881441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184bd643-6a0d-4d12-8e10-dfb0957cd748 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:20.881752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184bd643-6a0d-4d12-8e10-dfb0957cd748 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:20.883925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184bd643-6a0d-4d12-8e10-dfb0957cd748 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:20.889515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 388/774] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 2 00:42:20 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:20.895413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184bd643-6a0d-4d12-8e10-dfb0957cd748 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:21.000827 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:21.006930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184bd643-6a0d-4d12-8e10-dfb0957cd748 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:21.053970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccb8425-fe71-4aa3-9486-131d9ae23834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:21.054621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccb8425-fe71-4aa3-9486-131d9ae23834 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:21.054900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccb8425-fe71-4aa3-9486-131d9ae23834 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:21.055172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccb8425-fe71-4aa3-9486-131d9ae23834 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:21.055949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccb8425-fe71-4aa3-9486-131d9ae23834 None admin] RBAC: Adding query filter params (interface=public, service_id=14b7413954754fd79216a63e4f6d82b6, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:21.056257 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccb8425-fe71-4aa3-9486-131d9ae23834 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:21.057616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccb8425-fe71-4aa3-9486-131d9ae23834 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:21.066265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 389/775] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 2 00:42:20 2022] GET /identity/v3/endpoints?service_id=14b7413954754fd79216a63e4f6d82b6&interface=public®ion_id=RegionOne => generated 194 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:21.072367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccb8425-fe71-4aa3-9486-131d9ae23834 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:21.180712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccb8425-fe71-4aa3-9486-131d9ae23834 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:21.220094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762eadd8-8d5f-47cd-83d9-6cd8a1c6cc1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:21.220724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762eadd8-8d5f-47cd-83d9-6cd8a1c6cc1b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:21.221000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762eadd8-8d5f-47cd-83d9-6cd8a1c6cc1b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:21.221268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762eadd8-8d5f-47cd-83d9-6cd8a1c6cc1b None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:21.221881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762eadd8-8d5f-47cd-83d9-6cd8a1c6cc1b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:21.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762eadd8-8d5f-47cd-83d9-6cd8a1c6cc1b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:21.230780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 390/776] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 2 00:42:21 2022] GET /identity/v3/services => generated 772 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:21.245446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28ce113-3725-4324-8266-056b9168f075 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:21.284709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d0f05a-18ea-4d72-975d-b9ca091400bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:21.285305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d0f05a-18ea-4d72-975d-b9ca091400bd None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:21.285590 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d0f05a-18ea-4d72-975d-b9ca091400bd None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:21.285859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d0f05a-18ea-4d72-975d-b9ca091400bd None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:21.286535 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d0f05a-18ea-4d72-975d-b9ca091400bd None admin] RBAC: Adding query filter params (name=network) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:21.286838 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d0f05a-18ea-4d72-975d-b9ca091400bd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:21.288851 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d0f05a-18ea-4d72-975d-b9ca091400bd None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:21.294584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 387/777] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:20 2022] GET /identity/v3/services?name=network => generated 124 bytes in 432 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:21.300514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762eadd8-8d5f-47cd-83d9-6cd8a1c6cc1b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:21.343977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4c93d1-8a56-4a32-a8b4-6fd4b7a7981b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:21.344608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4c93d1-8a56-4a32-a8b4-6fd4b7a7981b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:21.345087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4c93d1-8a56-4a32-a8b4-6fd4b7a7981b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:21.348219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 388/778] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:21.351616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:21.354668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eff5f7-4e1c-46e1-9d47-1d4e565be8d9 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:21.355336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eff5f7-4e1c-46e1-9d47-1d4e565be8d9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:21.355823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6eff5f7-4e1c-46e1-9d47-1d4e565be8d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:21.396131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e6eff5f7-4e1c-46e1-9d47-1d4e565be8d9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:21.398046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e6eff5f7-4e1c-46e1-9d47-1d4e565be8d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:21.423058 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:21.453187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762eadd8-8d5f-47cd-83d9-6cd8a1c6cc1b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:21.495778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 389/779] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:21 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:21.497080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf55c56-f971-4442-9b9c-782870ef913a None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:21.497367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf55c56-f971-4442-9b9c-782870ef913a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:21.497639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf55c56-f971-4442-9b9c-782870ef913a None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:21.499248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 390/780] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:21.504879 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cb9815-0d80-48c9-9e8e-133b26728139 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:21.505234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cb9815-0d80-48c9-9e8e-133b26728139 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:21.505491 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506aae70-9db5-4918-b70d-6e5c6160c7ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:21.505620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cb9815-0d80-48c9-9e8e-133b26728139 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:21.506957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506aae70-9db5-4918-b70d-6e5c6160c7ab None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:21.507374 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506aae70-9db5-4918-b70d-6e5c6160c7ab None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:21.508027 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506aae70-9db5-4918-b70d-6e5c6160c7ab None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:21.508787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506aae70-9db5-4918-b70d-6e5c6160c7ab None admin] RBAC: Adding query filter params (type=network) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:21.509225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506aae70-9db5-4918-b70d-6e5c6160c7ab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:21.510904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506aae70-9db5-4918-b70d-6e5c6160c7ab None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:21.517903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 391/781] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:21 2022] GET /identity/v3/services?type=network => generated 350 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:21.526097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506aae70-9db5-4918-b70d-6e5c6160c7ab None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:21.533424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-d8cb9815-0d80-48c9-9e8e-133b26728139 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:21.534689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d8cb9815-0d80-48c9-9e8e-133b26728139 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:21.667795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 391/782] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:21 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:21.689180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:21.705699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506aae70-9db5-4918-b70d-6e5c6160c7ab None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:21.752944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20709993-9e54-4dd3-9826-b5d25837df7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:21.753793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20709993-9e54-4dd3-9826-b5d25837df7f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:21.754281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20709993-9e54-4dd3-9826-b5d25837df7f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:21.754699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20709993-9e54-4dd3-9826-b5d25837df7f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:21.755536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20709993-9e54-4dd3-9826-b5d25837df7f None admin] RBAC: Adding query filter params (interface=public, service_id=3544a5b656194de3a5c5ac7dae01035a, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:21.756507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20709993-9e54-4dd3-9826-b5d25837df7f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:21.758332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20709993-9e54-4dd3-9826-b5d25837df7f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:21.764996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 392/783] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 2 00:42:21 2022] GET /identity/v3/endpoints?service_id=3544a5b656194de3a5c5ac7dae01035a&interface=public®ion_id=RegionOne => generated 194 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:21.770459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cb9815-0d80-48c9-9e8e-133b26728139 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:21.845153 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20709993-9e54-4dd3-9826-b5d25837df7f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:21.883444 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:21.923899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8cb9815-0d80-48c9-9e8e-133b26728139 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:21.985592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927dbb79-5156-4acb-8fef-d39068fec233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:21.986343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927dbb79-5156-4acb-8fef-d39068fec233 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:21.986684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927dbb79-5156-4acb-8fef-d39068fec233 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:21.987065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927dbb79-5156-4acb-8fef-d39068fec233 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:21.987741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927dbb79-5156-4acb-8fef-d39068fec233 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:21.989195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927dbb79-5156-4acb-8fef-d39068fec233 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:21.995250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 392/784] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 2 00:42:21 2022] GET /identity/v3/services => generated 772 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:21.996146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927dbb79-5156-4acb-8fef-d39068fec233 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:22.013322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20709993-9e54-4dd3-9826-b5d25837df7f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:22.081371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2d2f49-f281-45c4-8142-5e774890ed13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:22.082500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2d2f49-f281-45c4-8142-5e774890ed13 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:22.082982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2d2f49-f281-45c4-8142-5e774890ed13 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:22.083446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2d2f49-f281-45c4-8142-5e774890ed13 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:22.111267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2d2f49-f281-45c4-8142-5e774890ed13 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:22.117092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927dbb79-5156-4acb-8fef-d39068fec233 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7e2d2f49-f281-45c4-8142-5e774890ed13 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:22.118267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:22.119190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:22.120037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:22.132836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 393/785] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:22.138293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2d2f49-f281-45c4-8142-5e774890ed13 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:22.156016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd3b18f-ce27-43c5-b2da-4099c0f17eaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:22.156676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3b18f-ce27-43c5-b2da-4099c0f17eaa None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:22.156958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3b18f-ce27-43c5-b2da-4099c0f17eaa None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:22.157235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd3b18f-ce27-43c5-b2da-4099c0f17eaa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:22.177471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:22.179217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd3b18f-ce27-43c5-b2da-4099c0f17eaa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-6cd3b18f-ce27-43c5-b2da-4099c0f17eaa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:22.185758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:22.187242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:22.188086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 393/786] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:22.203318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd3b18f-ce27-43c5-b2da-4099c0f17eaa None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:22.254443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:22.304803 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2d2f49-f281-45c4-8142-5e774890ed13 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:22.330508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd3b18f-ce27-43c5-b2da-4099c0f17eaa None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:22.370621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797a20ca-94e3-4525-8f45-a1909294acc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:22.371529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797a20ca-94e3-4525-8f45-a1909294acc7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:22.372181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797a20ca-94e3-4525-8f45-a1909294acc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:22.373596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797a20ca-94e3-4525-8f45-a1909294acc7 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:22.374630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797a20ca-94e3-4525-8f45-a1909294acc7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:22.376097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797a20ca-94e3-4525-8f45-a1909294acc7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:22.377643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797a20ca-94e3-4525-8f45-a1909294acc7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:22.394322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 394/787] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:42:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:22.398972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797a20ca-94e3-4525-8f45-a1909294acc7 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:22.402733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049911de-fe4c-446b-95a1-46772a076b7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:22.403553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049911de-fe4c-446b-95a1-46772a076b7d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:22.404005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049911de-fe4c-446b-95a1-46772a076b7d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:22.405253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049911de-fe4c-446b-95a1-46772a076b7d None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:22.406115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049911de-fe4c-446b-95a1-46772a076b7d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:22.408535 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049911de-fe4c-446b-95a1-46772a076b7d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:22.410112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049911de-fe4c-446b-95a1-46772a076b7d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:22.428168 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 394/788] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:42:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:22.432958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049911de-fe4c-446b-95a1-46772a076b7d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:22.541227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:22.544025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:22.585635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797a20ca-94e3-4525-8f45-a1909294acc7 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:22.608542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049911de-fe4c-446b-95a1-46772a076b7d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:22.647926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15eb93f-eb2f-4569-b29f-11d53b57fb35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:22.648856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15eb93f-eb2f-4569-b29f-11d53b57fb35 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:22.650010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15eb93f-eb2f-4569-b29f-11d53b57fb35 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:22.651796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15eb93f-eb2f-4569-b29f-11d53b57fb35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:22.669789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fe9609-55f1-498c-b2dd-5b6c145a9045 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:22.670906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15eb93f-eb2f-4569-b29f-11d53b57fb35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:22.671352 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fe9609-55f1-498c-b2dd-5b6c145a9045 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:22.671876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fe9609-55f1-498c-b2dd-5b6c145a9045 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:22.672318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fe9609-55f1-498c-b2dd-5b6c145a9045 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:22.672421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15eb93f-eb2f-4569-b29f-11d53b57fb35 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:22.686812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fe9609-55f1-498c-b2dd-5b6c145a9045 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:22.688566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fe9609-55f1-498c-b2dd-5b6c145a9045 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c15eb93f-eb2f-4569-b29f-11d53b57fb35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:22.702085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:22.703063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:22.704040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:22.704040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:22.704040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:22.704040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 395/789] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:22.706774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15eb93f-eb2f-4569-b29f-11d53b57fb35 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-95fe9609-55f1-498c-b2dd-5b6c145a9045 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:22.713017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:22.713962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:22.714900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:22.714900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:22.714900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:22.722213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 395/790] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:22.725131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fe9609-55f1-498c-b2dd-5b6c145a9045 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:22.895697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15eb93f-eb2f-4569-b29f-11d53b57fb35 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:22.902431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fe9609-55f1-498c-b2dd-5b6c145a9045 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:22.920263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:22.927226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:22.938688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbce065-43f9-49df-8dbf-7b125a492ef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:22.939226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbce065-43f9-49df-8dbf-7b125a492ef7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:22.939499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbce065-43f9-49df-8dbf-7b125a492ef7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:22.939771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbce065-43f9-49df-8dbf-7b125a492ef7 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:22.940400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbce065-43f9-49df-8dbf-7b125a492ef7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:22.940694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbce065-43f9-49df-8dbf-7b125a492ef7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:22.941630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7db95e-00cf-4c86-94ce-4313984c7d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:22.942175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbce065-43f9-49df-8dbf-7b125a492ef7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:22.942689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7db95e-00cf-4c86-94ce-4313984c7d0e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:22.943215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7db95e-00cf-4c86-94ce-4313984c7d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:22.944279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7db95e-00cf-4c86-94ce-4313984c7d0e None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:22.945245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7db95e-00cf-4c86-94ce-4313984c7d0e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:22.945621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7db95e-00cf-4c86-94ce-4313984c7d0e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:22.947062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7db95e-00cf-4c86-94ce-4313984c7d0e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:22.961897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 396/791] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:22.961897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e01a552-2998-4d75-8471-b07bcc053c12 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:22.961897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e01a552-2998-4d75-8471-b07bcc053c12 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:22.961897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e01a552-2998-4d75-8471-b07bcc053c12 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:22.962306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 397/792] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:22.964836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 396/793] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:22.965041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e01a552-2998-4d75-8471-b07bcc053c12 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:22.970981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7db95e-00cf-4c86-94ce-4313984c7d0e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:23.088437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7db95e-00cf-4c86-94ce-4313984c7d0e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:23.094172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e01a552-2998-4d75-8471-b07bcc053c12 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:23.131927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f92b0cc-b717-40a2-a74a-2780abf26c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:23.133206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f92b0cc-b717-40a2-a74a-2780abf26c46 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.133684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f92b0cc-b717-40a2-a74a-2780abf26c46 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.134035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbadd8f-1b7e-49a9-9702-91d3b304ece6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:23.134452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f92b0cc-b717-40a2-a74a-2780abf26c46 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.134690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbadd8f-1b7e-49a9-9702-91d3b304ece6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.134965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbadd8f-1b7e-49a9-9702-91d3b304ece6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.135232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbadd8f-1b7e-49a9-9702-91d3b304ece6 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.142831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f92b0cc-b717-40a2-a74a-2780abf26c46 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:23.143538 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbadd8f-1b7e-49a9-9702-91d3b304ece6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:23.144792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f92b0cc-b717-40a2-a74a-2780abf26c46 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:23.145063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbadd8f-1b7e-49a9-9702-91d3b304ece6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-1f92b0cc-b717-40a2-a74a-2780abf26c46 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:23.157795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:23.159031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:23.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:23.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:23.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:23.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:23.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:23.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:23.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:23.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:23.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:42:23.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-dbbadd8f-1b7e-49a9-9702-91d3b304ece6 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:23.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:23.161639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:23.162671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:23.162671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:23.162671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:23.162671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:23.162671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:23.162671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:23.162671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:23.162671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:23.162671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:23.162671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:23.169273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 398/795] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 2 00:42:22 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:23.169273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd00f860-152b-47ae-a821-ca9d87de4bff None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.169273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd00f860-152b-47ae-a821-ca9d87de4bff None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.169273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd00f860-152b-47ae-a821-ca9d87de4bff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.169761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 397/795] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 2 00:42:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:23.170981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f92b0cc-b717-40a2-a74a-2780abf26c46 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:23.187331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bd00f860-152b-47ae-a821-ca9d87de4bff None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:23.188612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bd00f860-152b-47ae-a821-ca9d87de4bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:23.214214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:23.240994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:23.268579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 399/796] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:23 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:23.269799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd00f860-152b-47ae-a821-ca9d87de4bff None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:23.291203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f92b0cc-b717-40a2-a74a-2780abf26c46 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:23.348133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05060045-153b-4b72-bc77-9bda364c7d43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:23.348842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05060045-153b-4b72-bc77-9bda364c7d43 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.349188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05060045-153b-4b72-bc77-9bda364c7d43 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.350045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05060045-153b-4b72-bc77-9bda364c7d43 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.350868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05060045-153b-4b72-bc77-9bda364c7d43 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:23.351225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05060045-153b-4b72-bc77-9bda364c7d43 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:23.352665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05060045-153b-4b72-bc77-9bda364c7d43 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:23.379370 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd00f860-152b-47ae-a821-ca9d87de4bff None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:23.384835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 398/797] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Oct 2 00:42:23 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:23.401257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05060045-153b-4b72-bc77-9bda364c7d43 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:23.424451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8f9f57-6490-4849-be43-9afc8a443b72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:23.425365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8f9f57-6490-4849-be43-9afc8a443b72 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.425887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8f9f57-6490-4849-be43-9afc8a443b72 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.426352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8f9f57-6490-4849-be43-9afc8a443b72 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.427266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8f9f57-6490-4849-be43-9afc8a443b72 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:23.427793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8f9f57-6490-4849-be43-9afc8a443b72 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:23.429427 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8f9f57-6490-4849-be43-9afc8a443b72 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:23.449214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 400/798] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 2 00:42:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:23.459275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8f9f57-6490-4849-be43-9afc8a443b72 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:23.501092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:23.525030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:23.531769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05060045-153b-4b72-bc77-9bda364c7d43 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:23.579642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8f9f57-6490-4849-be43-9afc8a443b72 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:23.618351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41765c39-a953-4e82-a3fa-6f50cd567425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:23.619367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41765c39-a953-4e82-a3fa-6f50cd567425 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.619884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41765c39-a953-4e82-a3fa-6f50cd567425 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.620318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41765c39-a953-4e82-a3fa-6f50cd567425 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.634766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4f3c4-8098-4f99-bd17-d829f0ceb891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:23.635505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4f3c4-8098-4f99-bd17-d829f0ceb891 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.635854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4f3c4-8098-4f99-bd17-d829f0ceb891 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.636200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4f3c4-8098-4f99-bd17-d829f0ceb891 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.652616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4f3c4-8098-4f99-bd17-d829f0ceb891 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:23.654121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4f3c4-8098-4f99-bd17-d829f0ceb891 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-faa4f3c4-8098-4f99-bd17-d829f0ceb891 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:23.670511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:23.671585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:23.672473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:23.672473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:23.672473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:23.672473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 399/799] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:23.672473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4f3c4-8098-4f99-bd17-d829f0ceb891 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:23.706611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41765c39-a953-4e82-a3fa-6f50cd567425 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:23.708050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41765c39-a953-4e82-a3fa-6f50cd567425 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:23.742753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:23.780105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4f3c4-8098-4f99-bd17-d829f0ceb891 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:23.812399 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 401/800] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:23 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:42:23.813658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cf18df-94a1-47e0-adad-afd4badac436 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.813942 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cf18df-94a1-47e0-adad-afd4badac436 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.814237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cf18df-94a1-47e0-adad-afd4badac436 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.818298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 402/801] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:23.818298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cf18df-94a1-47e0-adad-afd4badac436 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:23.821066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e43690a-2c69-45c0-9129-8b9db134bf9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:23.821689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e43690a-2c69-45c0-9129-8b9db134bf9e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.821961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e43690a-2c69-45c0-9129-8b9db134bf9e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.822251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e43690a-2c69-45c0-9129-8b9db134bf9e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.832623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e43690a-2c69-45c0-9129-8b9db134bf9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-1e43690a-2c69-45c0-9129-8b9db134bf9e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:23.838382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:23.839291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:23.840113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 400/802] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:23.840800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e43690a-2c69-45c0-9129-8b9db134bf9e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:23.852041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:23.925606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cf18df-94a1-47e0-adad-afd4badac436 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:23.948675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e43690a-2c69-45c0-9129-8b9db134bf9e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:23.957592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8036418f-830e-4c2e-ae60-c7566cf87a1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:23.958227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8036418f-830e-4c2e-ae60-c7566cf87a1f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.958501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8036418f-830e-4c2e-ae60-c7566cf87a1f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.958802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8036418f-830e-4c2e-ae60-c7566cf87a1f None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.959428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8036418f-830e-4c2e-ae60-c7566cf87a1f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:23.959728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8036418f-830e-4c2e-ae60-c7566cf87a1f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:23.961076 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8036418f-830e-4c2e-ae60-c7566cf87a1f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:23.963304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:23.972863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 403/803] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:23.973953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb9a6f-b42f-42d8-afcd-0cb2692c45e0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.974252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb9a6f-b42f-42d8-afcd-0cb2692c45e0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.975059 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eb9a6f-b42f-42d8-afcd-0cb2692c45e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.991258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40769157-bad3-4cd6-bde0-1e57cbf6b0e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:23.992136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-89eb9a6f-b42f-42d8-afcd-0cb2692c45e0 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:23.992251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40769157-bad3-4cd6-bde0-1e57cbf6b0e3 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:23.992743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40769157-bad3-4cd6-bde0-1e57cbf6b0e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:23.993211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40769157-bad3-4cd6-bde0-1e57cbf6b0e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:23.993314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-89eb9a6f-b42f-42d8-afcd-0cb2692c45e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:24.005065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40769157-bad3-4cd6-bde0-1e57cbf6b0e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.007907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40769157-bad3-4cd6-bde0-1e57cbf6b0e3 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-40769157-bad3-4cd6-bde0-1e57cbf6b0e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:24.023397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:24.024379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:24.025304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:24.025304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:24.025304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:24.025304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 401/804] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:24.025304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40769157-bad3-4cd6-bde0-1e57cbf6b0e3 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:24.072747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 404/805] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:23 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:24.073428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb9a6f-b42f-42d8-afcd-0cb2692c45e0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:24.118421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:24.175293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40769157-bad3-4cd6-bde0-1e57cbf6b0e3 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:24.175804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eb9a6f-b42f-42d8-afcd-0cb2692c45e0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:24.203272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:24.213445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b3bd9b-ae35-4f1e-b1de-95d8ad3776b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:24.213998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b3bd9b-ae35-4f1e-b1de-95d8ad3776b8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.214497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b3bd9b-ae35-4f1e-b1de-95d8ad3776b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.214854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b3bd9b-ae35-4f1e-b1de-95d8ad3776b8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.221668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b3bd9b-ae35-4f1e-b1de-95d8ad3776b8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.222794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961a01d7-b2c5-44db-af4a-7c54e64c7298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:24.223402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93b3bd9b-ae35-4f1e-b1de-95d8ad3776b8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.223734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961a01d7-b2c5-44db-af4a-7c54e64c7298 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.224825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961a01d7-b2c5-44db-af4a-7c54e64c7298 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.225242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961a01d7-b2c5-44db-af4a-7c54e64c7298 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.226253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961a01d7-b2c5-44db-af4a-7c54e64c7298 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:24.226700 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961a01d7-b2c5-44db-af4a-7c54e64c7298 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.228504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961a01d7-b2c5-44db-af4a-7c54e64c7298 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-93b3bd9b-ae35-4f1e-b1de-95d8ad3776b8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:24.235865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:24.236947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:24.238040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:24.238040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:24.238040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:24.238040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:24.238040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:24.238040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:24.238040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:24.238040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:24.244766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 405/806] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:24.245292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b3bd9b-ae35-4f1e-b1de-95d8ad3776b8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:24.245622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 402/807] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:42:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:24.246711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961a01d7-b2c5-44db-af4a-7c54e64c7298 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:24.359586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961a01d7-b2c5-44db-af4a-7c54e64c7298 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:24.370015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b3bd9b-ae35-4f1e-b1de-95d8ad3776b8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:24.383744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:24.397738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611ad3dd-0a30-44c7-8028-9ce8f71ce405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:24.398359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611ad3dd-0a30-44c7-8028-9ce8f71ce405 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.398748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611ad3dd-0a30-44c7-8028-9ce8f71ce405 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.399020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611ad3dd-0a30-44c7-8028-9ce8f71ce405 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.399819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611ad3dd-0a30-44c7-8028-9ce8f71ce405 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:24.400171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611ad3dd-0a30-44c7-8028-9ce8f71ce405 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.401636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611ad3dd-0a30-44c7-8028-9ce8f71ce405 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.406993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a369a398-4d6a-4785-83c7-c8d8b0edff80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:24.407600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369a398-4d6a-4785-83c7-c8d8b0edff80 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.407927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369a398-4d6a-4785-83c7-c8d8b0edff80 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.408202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369a398-4d6a-4785-83c7-c8d8b0edff80 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.409295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a369a398-4d6a-4785-83c7-c8d8b0edff80 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:24.409605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a369a398-4d6a-4785-83c7-c8d8b0edff80 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.411442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a369a398-4d6a-4785-83c7-c8d8b0edff80 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.418514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 403/808] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:24.418514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611ad3dd-0a30-44c7-8028-9ce8f71ce405 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:24.423785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 406/809] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:24.424463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a369a398-4d6a-4785-83c7-c8d8b0edff80 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:24.477039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:24.531534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611ad3dd-0a30-44c7-8028-9ce8f71ce405 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:24.532649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a369a398-4d6a-4785-83c7-c8d8b0edff80 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:24.566576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86bc5cb-7ee9-4eb0-a46c-15312c2b0714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:24.567176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb510849-f36f-419d-9fde-7be7e3e43d71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:24.567289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86bc5cb-7ee9-4eb0-a46c-15312c2b0714 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.567455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86bc5cb-7ee9-4eb0-a46c-15312c2b0714 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.567723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86bc5cb-7ee9-4eb0-a46c-15312c2b0714 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.567839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb510849-f36f-419d-9fde-7be7e3e43d71 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.568709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb510849-f36f-419d-9fde-7be7e3e43d71 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.568994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb510849-f36f-419d-9fde-7be7e3e43d71 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.569579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb510849-f36f-419d-9fde-7be7e3e43d71 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.570975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb510849-f36f-419d-9fde-7be7e3e43d71 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.578002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86bc5cb-7ee9-4eb0-a46c-15312c2b0714 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.579424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86bc5cb-7ee9-4eb0-a46c-15312c2b0714 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-eb510849-f36f-419d-9fde-7be7e3e43d71 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:24.579639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:24.580608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:24.581442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 407/810] 10.222.0.25 () {56 vars in 1129 bytes} [Sun Oct 2 00:42:24 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:24.582278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb510849-f36f-419d-9fde-7be7e3e43d71 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-f86bc5cb-7ee9-4eb0-a46c-15312c2b0714 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:24.595096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:24.596273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:24.597825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:24.597825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:24.597825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:24.597825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 404/811] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:24.597825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86bc5cb-7ee9-4eb0-a46c-15312c2b0714 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:24.635224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:24.684343 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb510849-f36f-419d-9fde-7be7e3e43d71 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:24.694773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:24.700590 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86bc5cb-7ee9-4eb0-a46c-15312c2b0714 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:24.719631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f1737-728d-44d9-a1f4-2b50eb72779c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:24.720276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f1737-728d-44d9-a1f4-2b50eb72779c None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.720553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f1737-728d-44d9-a1f4-2b50eb72779c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.720822 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f1737-728d-44d9-a1f4-2b50eb72779c None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/af51bf378ea2491780250fc037fa2920/roles/2c1bcaa088914d4593bb1debe2fc88d6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.734620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8fe01c-f05c-47b0-aae3-2e4dd42f211a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:24.734815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79f1737-728d-44d9-a1f4-2b50eb72779c None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=af51bf378ea2491780250fc037fa2920, role_id=2c1bcaa088914d4593bb1debe2fc88d6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.735239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8fe01c-f05c-47b0-aae3-2e4dd42f211a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.735516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8fe01c-f05c-47b0-aae3-2e4dd42f211a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.735789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8fe01c-f05c-47b0-aae3-2e4dd42f211a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.736283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79f1737-728d-44d9-a1f4-2b50eb72779c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.743375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8fe01c-f05c-47b0-aae3-2e4dd42f211a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.745098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8fe01c-f05c-47b0-aae3-2e4dd42f211a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.748461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 408/812] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:42:24 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/af51bf378ea2491780250fc037fa2920/roles/2c1bcaa088914d4593bb1debe2fc88d6 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:42:24.749630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63353e1a-eda2-47a0-a09d-9599ba5762ac None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.749907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63353e1a-eda2-47a0-a09d-9599ba5762ac None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.750205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63353e1a-eda2-47a0-a09d-9599ba5762ac None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.751421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 409/813] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:24.752134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63353e1a-eda2-47a0-a09d-9599ba5762ac None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-2f8fe01c-f05c-47b0-aae3-2e4dd42f211a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:24.760345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:24.761510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:24.762613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:24.762613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:24.762613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:24.762613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:24.762613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:24.762613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:24.762613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:24.762613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:24.762613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 405/814] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:24.763207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8fe01c-f05c-47b0-aae3-2e4dd42f211a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:24.874566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:24.880188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63353e1a-eda2-47a0-a09d-9599ba5762ac None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:24.883928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8fe01c-f05c-47b0-aae3-2e4dd42f211a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:24.920550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0992f07d-fb95-492b-86e8-915c98f711bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:24.921392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0992f07d-fb95-492b-86e8-915c98f711bc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.921793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0992f07d-fb95-492b-86e8-915c98f711bc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.922220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0992f07d-fb95-492b-86e8-915c98f711bc None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.922967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0992f07d-fb95-492b-86e8-915c98f711bc None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:24.923403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0992f07d-fb95-492b-86e8-915c98f711bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.924922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0992f07d-fb95-492b-86e8-915c98f711bc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.926224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a368df-5d6a-430d-bfd4-ec5edd23260f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:24.927024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a368df-5d6a-430d-bfd4-ec5edd23260f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.927859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a368df-5d6a-430d-bfd4-ec5edd23260f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.927859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a368df-5d6a-430d-bfd4-ec5edd23260f None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.928545 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a368df-5d6a-430d-bfd4-ec5edd23260f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:24.929646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a368df-5d6a-430d-bfd4-ec5edd23260f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:24.930422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a368df-5d6a-430d-bfd4-ec5edd23260f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:24.931289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 410/815] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 2 00:42:24 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:24.932774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f64cb8-dfbf-4af5-8794-9919fbcdac0f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:24.933048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f64cb8-dfbf-4af5-8794-9919fbcdac0f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:24.933318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f64cb8-dfbf-4af5-8794-9919fbcdac0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:24.952990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-16f64cb8-dfbf-4af5-8794-9919fbcdac0f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:24.954268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-16f64cb8-dfbf-4af5-8794-9919fbcdac0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:24.973788 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:25.045151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 411/816] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:24 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:25.045151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f64cb8-dfbf-4af5-8794-9919fbcdac0f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:25.064616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 406/817] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:25.065703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a368df-5d6a-430d-bfd4-ec5edd23260f None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:25.160490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f64cb8-dfbf-4af5-8794-9919fbcdac0f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:25.188358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a368df-5d6a-430d-bfd4-ec5edd23260f None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:25.196444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419e6973-d625-4076-aaed-0b8c66986f1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:25.197252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419e6973-d625-4076-aaed-0b8c66986f1f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:25.197654 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419e6973-d625-4076-aaed-0b8c66986f1f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:25.198209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419e6973-d625-4076-aaed-0b8c66986f1f None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:25.199092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419e6973-d625-4076-aaed-0b8c66986f1f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:25.199490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419e6973-d625-4076-aaed-0b8c66986f1f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:25.201192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419e6973-d625-4076-aaed-0b8c66986f1f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:25.214705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 412/818] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:25.215740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419e6973-d625-4076-aaed-0b8c66986f1f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:25.227608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3600b36a-619e-4c35-bef1-78c766d1c0e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:25.228211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3600b36a-619e-4c35-bef1-78c766d1c0e4 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:25.228487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3600b36a-619e-4c35-bef1-78c766d1c0e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:25.228767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3600b36a-619e-4c35-bef1-78c766d1c0e4 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:25.229765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3600b36a-619e-4c35-bef1-78c766d1c0e4 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:25.230092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3600b36a-619e-4c35-bef1-78c766d1c0e4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:25.232047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3600b36a-619e-4c35-bef1-78c766d1c0e4 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:25.235019 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:25.238315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 407/819] 10.222.0.25 () {56 vars in 1124 bytes} [Sun Oct 2 00:42:25 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:25.239317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3600b36a-619e-4c35-bef1-78c766d1c0e4 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:25.265489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:25.349040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419e6973-d625-4076-aaed-0b8c66986f1f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:25.360570 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3600b36a-619e-4c35-bef1-78c766d1c0e4 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:25.383435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1498358-4ef8-4b85-9825-9ca68379cff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:25.384052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1498358-4ef8-4b85-9825-9ca68379cff0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:25.384331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1498358-4ef8-4b85-9825-9ca68379cff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:25.384596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1498358-4ef8-4b85-9825-9ca68379cff0 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:25.391948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1498358-4ef8-4b85-9825-9ca68379cff0 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:25.393372 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1498358-4ef8-4b85-9825-9ca68379cff0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:25.397258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72784a5b-9ac6-4259-b898-6736746eed12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:25.398195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72784a5b-9ac6-4259-b898-6736746eed12 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:25.398817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72784a5b-9ac6-4259-b898-6736746eed12 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:25.399290 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72784a5b-9ac6-4259-b898-6736746eed12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:25.400274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72784a5b-9ac6-4259-b898-6736746eed12 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=dad93d99b865407aa120035015f4692a) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:25.400811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72784a5b-9ac6-4259-b898-6736746eed12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:25.402464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72784a5b-9ac6-4259-b898-6736746eed12 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e1498358-4ef8-4b85-9825-9ca68379cff0 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:25.404898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:25.405798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 413/820] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:25 2022] GET /identity/v3/users/swift => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:25.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1498358-4ef8-4b85-9825-9ca68379cff0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:25.413668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 408/821] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:25 2022] GET /identity/v3/role_assignments?user.id=dad93d99b865407aa120035015f4692a&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:25.414876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72784a5b-9ac6-4259-b898-6736746eed12 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:25.496792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:25.515248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:25.526024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1498358-4ef8-4b85-9825-9ca68379cff0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:25.538047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72784a5b-9ac6-4259-b898-6736746eed12 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:25.563944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be701ef-87a5-48b2-8c4b-5fb625b5af3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:25.565012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be701ef-87a5-48b2-8c4b-5fb625b5af3d None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:25.565464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be701ef-87a5-48b2-8c4b-5fb625b5af3d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:25.565967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be701ef-87a5-48b2-8c4b-5fb625b5af3d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:25.566850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be701ef-87a5-48b2-8c4b-5fb625b5af3d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:25.569209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be701ef-87a5-48b2-8c4b-5fb625b5af3d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:25.578475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e062c7b-5927-413c-924d-71ad1c28791e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:25.579539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e062c7b-5927-413c-924d-71ad1c28791e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:25.580059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e062c7b-5927-413c-924d-71ad1c28791e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:25.580506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e062c7b-5927-413c-924d-71ad1c28791e None admin] PATH_INFO: `/v3/services/network` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:25.581383 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e062c7b-5927-413c-924d-71ad1c28791e None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:25.583066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e062c7b-5927-413c-924d-71ad1c28791e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:25.594573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 414/822] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:42:25 2022] POST /identity/v3/endpoints => generated 346 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-4e062c7b-5927-413c-924d-71ad1c28791e None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:25.594848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:25.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 02 00:42:25.596677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:25.597404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be701ef-87a5-48b2-8c4b-5fb625b5af3d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:25.600659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 409/823] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:25 2022] GET /identity/v3/services/network => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:25.611248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e062c7b-5927-413c-924d-71ad1c28791e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:25.752964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e062c7b-5927-413c-924d-71ad1c28791e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:25.786356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be701ef-87a5-48b2-8c4b-5fb625b5af3d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:25.797145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eeb2d7-b9c3-4805-b903-7a5ff75d34db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:25.797767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eeb2d7-b9c3-4805-b903-7a5ff75d34db None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:25.798042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eeb2d7-b9c3-4805-b903-7a5ff75d34db None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:25.798336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eeb2d7-b9c3-4805-b903-7a5ff75d34db None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:25.798967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eeb2d7-b9c3-4805-b903-7a5ff75d34db None admin] RBAC: Adding query filter params (name=network) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:25.799349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eeb2d7-b9c3-4805-b903-7a5ff75d34db None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:25.800738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eeb2d7-b9c3-4805-b903-7a5ff75d34db None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:25.806296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 410/824] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:25 2022] GET /identity/v3/services?name=network => generated 124 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:25.811809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eeb2d7-b9c3-4805-b903-7a5ff75d34db None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:25.820932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:25.824894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:25.836229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af728d0-eb59-475b-9be8-d439924a5b0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:25.836925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af728d0-eb59-475b-9be8-d439924a5b0e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:25.837261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af728d0-eb59-475b-9be8-d439924a5b0e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:25.837594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af728d0-eb59-475b-9be8-d439924a5b0e None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:25.838414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af728d0-eb59-475b-9be8-d439924a5b0e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:25.838784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af728d0-eb59-475b-9be8-d439924a5b0e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:25.840284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af728d0-eb59-475b-9be8-d439924a5b0e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:25.862503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 415/825] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 2 00:42:25 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:25.868208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af728d0-eb59-475b-9be8-d439924a5b0e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:25.947888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eeb2d7-b9c3-4805-b903-7a5ff75d34db None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:26.017061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae2950-9e70-4273-9450-791fc99cb35d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:26.017586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae2950-9e70-4273-9450-791fc99cb35d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:26.017862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae2950-9e70-4273-9450-791fc99cb35d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:26.018161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae2950-9e70-4273-9450-791fc99cb35d None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:26.018789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae2950-9e70-4273-9450-791fc99cb35d None admin] RBAC: Adding query filter params (type=network) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:26.019084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae2950-9e70-4273-9450-791fc99cb35d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:26.020439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae2950-9e70-4273-9450-791fc99cb35d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:26.024102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af728d0-eb59-475b-9be8-d439924a5b0e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:26.041573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 411/826] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:25 2022] GET /identity/v3/services?type=network => generated 350 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:26.047001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae2950-9e70-4273-9450-791fc99cb35d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:26.092092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406537c5-c57c-4c73-b751-18140ed21244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:26.092719 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406537c5-c57c-4c73-b751-18140ed21244 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:26.092994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406537c5-c57c-4c73-b751-18140ed21244 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:26.093263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406537c5-c57c-4c73-b751-18140ed21244 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:26.116650 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406537c5-c57c-4c73-b751-18140ed21244 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:26.118236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406537c5-c57c-4c73-b751-18140ed21244 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:26.130745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-406537c5-c57c-4c73-b751-18140ed21244 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:26.145582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:26.147148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:26.148039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:26.148039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:26.148039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:26.148039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 416/827] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:26.153167 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406537c5-c57c-4c73-b751-18140ed21244 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:26.173354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae2950-9e70-4273-9450-791fc99cb35d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:26.213923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:26.216536 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfa03c1-ebca-4d60-9328-b05023cd1992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:26.217195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfa03c1-ebca-4d60-9328-b05023cd1992 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:26.217472 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfa03c1-ebca-4d60-9328-b05023cd1992 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:26.217735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfa03c1-ebca-4d60-9328-b05023cd1992 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:26.218431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfa03c1-ebca-4d60-9328-b05023cd1992 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:26.221754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfa03c1-ebca-4d60-9328-b05023cd1992 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:26.260982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 412/828] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:42:26 2022] POST /identity/v3/endpoints => generated 337 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:26.310690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406537c5-c57c-4c73-b751-18140ed21244 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:26.373332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a248ea-6386-452b-b988-2cbceef64e67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:26.373967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a248ea-6386-452b-b988-2cbceef64e67 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:26.374256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a248ea-6386-452b-b988-2cbceef64e67 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:26.374527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a248ea-6386-452b-b988-2cbceef64e67 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:26.375174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a248ea-6386-452b-b988-2cbceef64e67 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:26.375472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a248ea-6386-452b-b988-2cbceef64e67 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:26.376955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a248ea-6386-452b-b988-2cbceef64e67 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:26.392326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 417/829] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:26.397898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfa03c1-ebca-4d60-9328-b05023cd1992 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:26.506743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfa03c1-ebca-4d60-9328-b05023cd1992 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:26.536240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:26.543799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9758583-ad93-4953-a31c-3378c03b42f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:26.544614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9758583-ad93-4953-a31c-3378c03b42f1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:26.545056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9758583-ad93-4953-a31c-3378c03b42f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:26.545331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9758583-ad93-4953-a31c-3378c03b42f1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:26.552615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9758583-ad93-4953-a31c-3378c03b42f1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:26.554049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9758583-ad93-4953-a31c-3378c03b42f1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-d9758583-ad93-4953-a31c-3378c03b42f1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:26.565890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:26.566836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:26.567727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:26.567727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:26.567727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:26.567727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:26.567727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:26.567727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:26.567727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:26.567727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:26.567727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 413/830] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:26.572664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a248ea-6386-452b-b988-2cbceef64e67 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:26.683906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:26.695190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a248ea-6386-452b-b988-2cbceef64e67 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:26.732592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9627430-1c9b-43d6-8ba9-900aa0118448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:26.733510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9627430-1c9b-43d6-8ba9-900aa0118448 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:26.733983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9627430-1c9b-43d6-8ba9-900aa0118448 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:26.736433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9627430-1c9b-43d6-8ba9-900aa0118448 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:26.737355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9627430-1c9b-43d6-8ba9-900aa0118448 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:26.737830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9627430-1c9b-43d6-8ba9-900aa0118448 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:26.739513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9627430-1c9b-43d6-8ba9-900aa0118448 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:26.755215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 418/831] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:26.761090 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9758583-ad93-4953-a31c-3378c03b42f1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:26.870423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9758583-ad93-4953-a31c-3378c03b42f1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:26.904589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2290d4f-5a78-4045-b617-39c2436cf74e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:26.905451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2290d4f-5a78-4045-b617-39c2436cf74e None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:26.905867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2290d4f-5a78-4045-b617-39c2436cf74e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:26.906369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2290d4f-5a78-4045-b617-39c2436cf74e None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/db6c9dbbcf244a6f8bac59d2c9c973bc/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:26.908357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2290d4f-5a78-4045-b617-39c2436cf74e None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=db6c9dbbcf244a6f8bac59d2c9c973bc, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:26.909933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2290d4f-5a78-4045-b617-39c2436cf74e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:26.924180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 414/832] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:42:26 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/db6c9dbbcf244a6f8bac59d2c9c973bc/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:42:27.213149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9c4e6b-d8cb-454f-9301-047edc78ec63 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:27.213542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9c4e6b-d8cb-454f-9301-047edc78ec63 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:27.213938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9c4e6b-d8cb-454f-9301-047edc78ec63 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:27.215638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 419/833] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:27.220849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d481d2-6b14-4e9c-a11b-de5342e46897 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:27.221287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d481d2-6b14-4e9c-a11b-de5342e46897 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:27.221575 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d481d2-6b14-4e9c-a11b-de5342e46897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:27.243418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-41d481d2-6b14-4e9c-a11b-de5342e46897 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:27.244822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-41d481d2-6b14-4e9c-a11b-de5342e46897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:27.292178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:27.313891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 415/834] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:27 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:27.582374 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9c4e6b-d8cb-454f-9301-047edc78ec63 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:27.692103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9c4e6b-d8cb-454f-9301-047edc78ec63 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:27.728830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3ac60e-8d65-4279-87a6-2968c065ab6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:27.729454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3ac60e-8d65-4279-87a6-2968c065ab6b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:27.729727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3ac60e-8d65-4279-87a6-2968c065ab6b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:27.729993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3ac60e-8d65-4279-87a6-2968c065ab6b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:27.741426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3ac60e-8d65-4279-87a6-2968c065ab6b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6d3ac60e-8d65-4279-87a6-2968c065ab6b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:27.746251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:27.747430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:27.748808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 420/835] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:42:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:27.758524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d481d2-6b14-4e9c-a11b-de5342e46897 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:27.860745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d481d2-6b14-4e9c-a11b-de5342e46897 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:27.894767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307ce1f5-bdef-4b09-b50a-e1879676317c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:27.895375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ce1f5-bdef-4b09-b50a-e1879676317c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:27.895645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ce1f5-bdef-4b09-b50a-e1879676317c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:27.895929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ce1f5-bdef-4b09-b50a-e1879676317c None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:27.896686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307ce1f5-bdef-4b09-b50a-e1879676317c None admin] RBAC: Adding query filter params (name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:27.896988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307ce1f5-bdef-4b09-b50a-e1879676317c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:27.898350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307ce1f5-bdef-4b09-b50a-e1879676317c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:27.906481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 416/836] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:42:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:27.915144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3ac60e-8d65-4279-87a6-2968c065ab6b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:27.932268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:28.029195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3ac60e-8d65-4279-87a6-2968c065ab6b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:28.069430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e160a6-91d6-4bfe-aa72-83547be47a3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:28.070086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e160a6-91d6-4bfe-aa72-83547be47a3e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.070371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e160a6-91d6-4bfe-aa72-83547be47a3e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.070638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e160a6-91d6-4bfe-aa72-83547be47a3e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.073846 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da20d96c-7f9a-43a8-92c4-49f4a94cab2d None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.074174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da20d96c-7f9a-43a8-92c4-49f4a94cab2d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.074465 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da20d96c-7f9a-43a8-92c4-49f4a94cab2d None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.077209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 417/837] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:28.081711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f875a7-5932-4a45-bcc5-0c681bf4874e None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.081994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f875a7-5932-4a45-bcc5-0c681bf4874e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.082185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e160a6-91d6-4bfe-aa72-83547be47a3e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:28.082348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f875a7-5932-4a45-bcc5-0c681bf4874e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.084168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e160a6-91d6-4bfe-aa72-83547be47a3e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:28.101893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a7f875a7-5932-4a45-bcc5-0c681bf4874e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-70e160a6-91d6-4bfe-aa72-83547be47a3e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:28.102119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:28.103003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:28.103908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:28.103908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:28.103908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:28.104103 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a7f875a7-5932-4a45-bcc5-0c681bf4874e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:28.104180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 421/838] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:28.109754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e160a6-91d6-4bfe-aa72-83547be47a3e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:28.141554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:28.176299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 418/839] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:28 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:28.187434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:28.197489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec255c4-d4f9-49ea-8d68-64eef8cfbd8e None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.197788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec255c4-d4f9-49ea-8d68-64eef8cfbd8e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.198030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec255c4-d4f9-49ea-8d68-64eef8cfbd8e None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.199492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 419/840] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:28.204807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89de154-48d9-4a51-ad9f-fc7852f6b75e None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.205325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89de154-48d9-4a51-ad9f-fc7852f6b75e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.205793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89de154-48d9-4a51-ad9f-fc7852f6b75e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.226112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f89de154-48d9-4a51-ad9f-fc7852f6b75e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:28.228009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f89de154-48d9-4a51-ad9f-fc7852f6b75e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:28.232120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e160a6-91d6-4bfe-aa72-83547be47a3e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:28.275699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec05430-d375-484f-a23a-a41bab8ac2a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:28.276545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec05430-d375-484f-a23a-a41bab8ac2a5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.276815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec05430-d375-484f-a23a-a41bab8ac2a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.277088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec05430-d375-484f-a23a-a41bab8ac2a5 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.277713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec05430-d375-484f-a23a-a41bab8ac2a5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:28.278021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec05430-d375-484f-a23a-a41bab8ac2a5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:28.279394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec05430-d375-484f-a23a-a41bab8ac2a5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:28.296084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 422/841] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:28.296966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec05430-d375-484f-a23a-a41bab8ac2a5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:28.313105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 420/842] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:28 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:28.397512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec05430-d375-484f-a23a-a41bab8ac2a5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:28.420721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:28.431989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aac2c1-8873-405d-a631-723614c7841a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:28.432767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aac2c1-8873-405d-a631-723614c7841a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.433182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aac2c1-8873-405d-a631-723614c7841a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.433548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aac2c1-8873-405d-a631-723614c7841a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.440793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39aac2c1-8873-405d-a631-723614c7841a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:28.442447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39aac2c1-8873-405d-a631-723614c7841a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:28.446002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89de154-48d9-4a51-ad9f-fc7852f6b75e None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-39aac2c1-8873-405d-a631-723614c7841a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:28.457865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:28.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:28.460176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:28.460176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:28.460176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:28.460176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:28.460176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:28.460176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:28.460176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:28.460176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:28.460176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:28.460176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:28.462621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 423/843] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 2 00:42:28 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:28.467639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aac2c1-8873-405d-a631-723614c7841a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:28.574199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89de154-48d9-4a51-ad9f-fc7852f6b75e None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:28.576490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:28.579009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aac2c1-8873-405d-a631-723614c7841a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:28.616197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e12844-535e-4667-9053-1c1bf8dbacf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:28.617026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3019ed3e-d099-4d1b-9873-6c7caee72bab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:28.617651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3019ed3e-d099-4d1b-9873-6c7caee72bab None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.617811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e12844-535e-4667-9053-1c1bf8dbacf5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.617915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3019ed3e-d099-4d1b-9873-6c7caee72bab None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.618208 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3019ed3e-d099-4d1b-9873-6c7caee72bab None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.618494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e12844-535e-4667-9053-1c1bf8dbacf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.618786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3019ed3e-d099-4d1b-9873-6c7caee72bab None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:28.619080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e12844-535e-4667-9053-1c1bf8dbacf5 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.619943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e12844-535e-4667-9053-1c1bf8dbacf5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:28.620133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3019ed3e-d099-4d1b-9873-6c7caee72bab None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:28.620540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e12844-535e-4667-9053-1c1bf8dbacf5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:28.622272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e12844-535e-4667-9053-1c1bf8dbacf5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-3019ed3e-d099-4d1b-9873-6c7caee72bab None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:28.628857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:28.635599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:28.636445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 421/844] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:42:28 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:28.637251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3019ed3e-d099-4d1b-9873-6c7caee72bab None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:28.649947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 424/845] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Oct 2 00:42:28 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:28.649947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e12844-535e-4667-9053-1c1bf8dbacf5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:28.694246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:28.740742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3019ed3e-d099-4d1b-9873-6c7caee72bab None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:28.759407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e12844-535e-4667-9053-1c1bf8dbacf5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:28.778695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eec84b9-50d6-4f96-baa0-bcf6f4880b3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:28.779330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eec84b9-50d6-4f96-baa0-bcf6f4880b3e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.779611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eec84b9-50d6-4f96-baa0-bcf6f4880b3e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.779884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eec84b9-50d6-4f96-baa0-bcf6f4880b3e None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.780880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eec84b9-50d6-4f96-baa0-bcf6f4880b3e None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:28.782262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eec84b9-50d6-4f96-baa0-bcf6f4880b3e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-2eec84b9-50d6-4f96-baa0-bcf6f4880b3e None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:28.792956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:28.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:28.794856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 422/846] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Oct 2 00:42:28 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:28.795649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eec84b9-50d6-4f96-baa0-bcf6f4880b3e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:28.800527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed45ff12-6071-45a5-bd91-3eb6e015ae36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:28.801144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed45ff12-6071-45a5-bd91-3eb6e015ae36 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.801420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed45ff12-6071-45a5-bd91-3eb6e015ae36 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.801688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed45ff12-6071-45a5-bd91-3eb6e015ae36 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.802488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed45ff12-6071-45a5-bd91-3eb6e015ae36 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:28.802791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed45ff12-6071-45a5-bd91-3eb6e015ae36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:28.804683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed45ff12-6071-45a5-bd91-3eb6e015ae36 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:28.811965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 425/847] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:42:28 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:28.812669 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed45ff12-6071-45a5-bd91-3eb6e015ae36 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:28.831197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:28.908494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eec84b9-50d6-4f96-baa0-bcf6f4880b3e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:28.929792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed45ff12-6071-45a5-bd91-3eb6e015ae36 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:28.937321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:28.944730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf773b-1ca3-4f2a-acce-345146c80dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:28.945333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf773b-1ca3-4f2a-acce-345146c80dc1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.945619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf773b-1ca3-4f2a-acce-345146c80dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.945884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edf773b-1ca3-4f2a-acce-345146c80dc1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.968379 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34e7ef-5d46-453c-9f1a-a600881fb2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:28.969178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34e7ef-5d46-453c-9f1a-a600881fb2d2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:28.969593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34e7ef-5d46-453c-9f1a-a600881fb2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:28.970012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34e7ef-5d46-453c-9f1a-a600881fb2d2 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:28.970656 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d34e7ef-5d46-453c-9f1a-a600881fb2d2 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:28.971134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d34e7ef-5d46-453c-9f1a-a600881fb2d2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:28.972287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d34e7ef-5d46-453c-9f1a-a600881fb2d2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:28.981408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 426/848] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:28 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:28.981408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34e7ef-5d46-453c-9f1a-a600881fb2d2 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:29.032040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edf773b-1ca3-4f2a-acce-345146c80dc1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:29.033526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edf773b-1ca3-4f2a-acce-345146c80dc1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:29.065089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:29.102557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34e7ef-5d46-453c-9f1a-a600881fb2d2 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:29.136055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd49cf65-aace-4b14-aa86-1d1fd775c899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:29.136657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd49cf65-aace-4b14-aa86-1d1fd775c899 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:29.136933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd49cf65-aace-4b14-aa86-1d1fd775c899 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:29.137202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd49cf65-aace-4b14-aa86-1d1fd775c899 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:29.137823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd49cf65-aace-4b14-aa86-1d1fd775c899 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:29.138148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd49cf65-aace-4b14-aa86-1d1fd775c899 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:29.139510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd49cf65-aace-4b14-aa86-1d1fd775c899 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:29.141963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 423/849] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:28 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:42:29.142816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf773b-1ca3-4f2a-acce-345146c80dc1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:29.149125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 427/850] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:42:28 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:29.150312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23490a4a-8d15-4158-b0dd-d6a6c81d9997 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:29.151071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23490a4a-8d15-4158-b0dd-d6a6c81d9997 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:29.151422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23490a4a-8d15-4158-b0dd-d6a6c81d9997 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:29.158119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 428/851] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:29.158119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23490a4a-8d15-4158-b0dd-d6a6c81d9997 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:29.195863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:29.264289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edf773b-1ca3-4f2a-acce-345146c80dc1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:29.269790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23490a4a-8d15-4158-b0dd-d6a6c81d9997 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:29.296091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:29.302119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c0cf7e-6f5a-4f35-87c5-8b90b0eebf8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:29.303216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c0cf7e-6f5a-4f35-87c5-8b90b0eebf8d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:29.303563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c0cf7e-6f5a-4f35-87c5-8b90b0eebf8d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:29.304097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c0cf7e-6f5a-4f35-87c5-8b90b0eebf8d None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:29.304742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c0cf7e-6f5a-4f35-87c5-8b90b0eebf8d None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:29.305201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c0cf7e-6f5a-4f35-87c5-8b90b0eebf8d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:29.307226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c0cf7e-6f5a-4f35-87c5-8b90b0eebf8d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:29.311314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f87f91-755d-4fc5-bfc9-3436a4c3a63f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:29.318454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f87f91-755d-4fc5-bfc9-3436a4c3a63f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:29.318782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 424/852] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:29 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:29.318893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f87f91-755d-4fc5-bfc9-3436a4c3a63f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:29.318893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f87f91-755d-4fc5-bfc9-3436a4c3a63f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:29.319050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0b98dc-7033-4ade-8620-677bc6d7fa26 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:29.319050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0b98dc-7033-4ade-8620-677bc6d7fa26 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:29.319050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0b98dc-7033-4ade-8620-677bc6d7fa26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:29.330797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f87f91-755d-4fc5-bfc9-3436a4c3a63f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:29.332281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f87f91-755d-4fc5-bfc9-3436a4c3a63f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:29.336508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8d0b98dc-7033-4ade-8620-677bc6d7fa26 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:29.337961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8d0b98dc-7033-4ade-8620-677bc6d7fa26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-45f87f91-755d-4fc5-bfc9-3436a4c3a63f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:29.351192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:29.352235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:29.353145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:29.353145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:29.353145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:29.353145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 429/853] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:29.358760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f87f91-755d-4fc5-bfc9-3436a4c3a63f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:29.442196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 425/854] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:29 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:29.477052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:29.494356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f87f91-755d-4fc5-bfc9-3436a4c3a63f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:29.535286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad32cb6-be3f-4eb7-94c4-2e4e89486de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:29.536308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad32cb6-be3f-4eb7-94c4-2e4e89486de1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:29.537354 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad32cb6-be3f-4eb7-94c4-2e4e89486de1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:29.537823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad32cb6-be3f-4eb7-94c4-2e4e89486de1 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:29.538711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad32cb6-be3f-4eb7-94c4-2e4e89486de1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:29.539212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad32cb6-be3f-4eb7-94c4-2e4e89486de1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:29.541824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad32cb6-be3f-4eb7-94c4-2e4e89486de1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:29.554851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 430/855] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:29.563158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0b98dc-7033-4ade-8620-677bc6d7fa26 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:29.672346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0b98dc-7033-4ade-8620-677bc6d7fa26 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:29.697587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:29.709794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea3bea5-7d20-4b1c-91e7-8e5063cd4ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:29.710732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea3bea5-7d20-4b1c-91e7-8e5063cd4ce0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:29.711233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea3bea5-7d20-4b1c-91e7-8e5063cd4ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:29.711702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea3bea5-7d20-4b1c-91e7-8e5063cd4ce0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:29.718783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea3bea5-7d20-4b1c-91e7-8e5063cd4ce0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:29.720435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea3bea5-7d20-4b1c-91e7-8e5063cd4ce0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-bea3bea5-7d20-4b1c-91e7-8e5063cd4ce0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:29.731927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:29.732796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:29.733678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:29.733678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:29.733678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:29.733678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:29.733678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:29.733678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:29.733678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:29.733678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:29.735985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 426/856] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:29.742291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad32cb6-be3f-4eb7-94c4-2e4e89486de1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:29.749607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea3bea5-7d20-4b1c-91e7-8e5063cd4ce0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:29.852315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad32cb6-be3f-4eb7-94c4-2e4e89486de1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:29.871425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea3bea5-7d20-4b1c-91e7-8e5063cd4ce0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:29.888476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7326cfc1-6532-4498-afe2-4e72b3ccef14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:29.889076 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7326cfc1-6532-4498-afe2-4e72b3ccef14 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:29.889349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7326cfc1-6532-4498-afe2-4e72b3ccef14 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:29.889616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7326cfc1-6532-4498-afe2-4e72b3ccef14 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:29.890327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7326cfc1-6532-4498-afe2-4e72b3ccef14 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:29.890628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7326cfc1-6532-4498-afe2-4e72b3ccef14 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:29.892105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7326cfc1-6532-4498-afe2-4e72b3ccef14 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:29.904402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 431/857] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:29.909985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7326cfc1-6532-4498-afe2-4e72b3ccef14 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:29.916585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc063b96-c705-4ca7-b540-de7633577039 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:29.917219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc063b96-c705-4ca7-b540-de7633577039 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:29.917493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc063b96-c705-4ca7-b540-de7633577039 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:29.917805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc063b96-c705-4ca7-b540-de7633577039 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:29.921542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:29.932273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc063b96-c705-4ca7-b540-de7633577039 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-dc063b96-c705-4ca7-b540-de7633577039 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:29.937946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:29.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:42:29.940421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:29.944200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 427/858] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:29.945450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc063b96-c705-4ca7-b540-de7633577039 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:30.003429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:30.024633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7326cfc1-6532-4498-afe2-4e72b3ccef14 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:30.063785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b4ab6c-37cf-486f-b45a-af7ee90b7223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:30.064389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b4ab6c-37cf-486f-b45a-af7ee90b7223 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:30.064664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b4ab6c-37cf-486f-b45a-af7ee90b7223 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:30.064930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b4ab6c-37cf-486f-b45a-af7ee90b7223 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:30.065649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b4ab6c-37cf-486f-b45a-af7ee90b7223 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=af51bf378ea2491780250fc037fa2920) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:30.065953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b4ab6c-37cf-486f-b45a-af7ee90b7223 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:30.067414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b4ab6c-37cf-486f-b45a-af7ee90b7223 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:30.074482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 432/859] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:29 2022] GET /identity/v3/role_assignments?user.id=af51bf378ea2491780250fc037fa2920&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:30.077517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc063b96-c705-4ca7-b540-de7633577039 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:30.121724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8bcf42-925b-4b48-8244-f3f4f27af906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:30.122423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8bcf42-925b-4b48-8244-f3f4f27af906 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:30.122729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8bcf42-925b-4b48-8244-f3f4f27af906 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:30.122964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8bcf42-925b-4b48-8244-f3f4f27af906 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:30.123627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8bcf42-925b-4b48-8244-f3f4f27af906 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:30.123927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8bcf42-925b-4b48-8244-f3f4f27af906 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:30.125289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8bcf42-925b-4b48-8244-f3f4f27af906 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:30.136393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 428/860] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:42:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:30.142340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b4ab6c-37cf-486f-b45a-af7ee90b7223 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:30.225472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:30.263980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b4ab6c-37cf-486f-b45a-af7ee90b7223 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:30.299715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffbb96c-0b77-4206-959e-4ff3ccb8a510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:30.300335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffbb96c-0b77-4206-959e-4ff3ccb8a510 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:30.300606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffbb96c-0b77-4206-959e-4ff3ccb8a510 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:30.300872 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffbb96c-0b77-4206-959e-4ff3ccb8a510 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:30.311696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffbb96c-0b77-4206-959e-4ff3ccb8a510 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:30.313116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffbb96c-0b77-4206-959e-4ff3ccb8a510 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-fffbb96c-0b77-4206-959e-4ff3ccb8a510 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:30.327165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:30.328118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:30.329439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:30.329439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:30.329439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:30.329439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 433/861] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:30.334772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8bcf42-925b-4b48-8244-f3f4f27af906 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:30.442251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8bcf42-925b-4b48-8244-f3f4f27af906 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:30.456842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:30.478247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb6046b-40e0-408b-90bb-565bd763e0f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:30.478859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb6046b-40e0-408b-90bb-565bd763e0f1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:30.479137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb6046b-40e0-408b-90bb-565bd763e0f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:30.479410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb6046b-40e0-408b-90bb-565bd763e0f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:30.480053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb6046b-40e0-408b-90bb-565bd763e0f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:30.480411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb6046b-40e0-408b-90bb-565bd763e0f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:30.481766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb6046b-40e0-408b-90bb-565bd763e0f1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:30.495011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 429/862] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:30.500643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffbb96c-0b77-4206-959e-4ff3ccb8a510 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:30.605822 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffbb96c-0b77-4206-959e-4ff3ccb8a510 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:30.629583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:30.642529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959d70ae-2cbd-43ca-aa33-68e97550cc45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:30.643162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959d70ae-2cbd-43ca-aa33-68e97550cc45 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:30.643487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959d70ae-2cbd-43ca-aa33-68e97550cc45 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:30.643770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959d70ae-2cbd-43ca-aa33-68e97550cc45 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:30.652680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959d70ae-2cbd-43ca-aa33-68e97550cc45 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:30.654365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959d70ae-2cbd-43ca-aa33-68e97550cc45 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-959d70ae-2cbd-43ca-aa33-68e97550cc45 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:30.666763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:30.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:30.668689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 434/863] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:42:30 2022] GET /identity/v3/users/swift => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:30.675442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb6046b-40e0-408b-90bb-565bd763e0f1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:30.783738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb6046b-40e0-408b-90bb-565bd763e0f1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:30.819562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68264cd5-5d7c-4cb0-9730-d1f8c665c2cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:30.820549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68264cd5-5d7c-4cb0-9730-d1f8c665c2cf None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:30.821062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68264cd5-5d7c-4cb0-9730-d1f8c665c2cf None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:30.821526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68264cd5-5d7c-4cb0-9730-d1f8c665c2cf None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:30.822484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68264cd5-5d7c-4cb0-9730-d1f8c665c2cf None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:30.822992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68264cd5-5d7c-4cb0-9730-d1f8c665c2cf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:30.824599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68264cd5-5d7c-4cb0-9730-d1f8c665c2cf None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:30.842555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 430/864] 10.222.0.25 () {56 vars in 1135 bytes} [Sun Oct 2 00:42:30 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:30.848498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959d70ae-2cbd-43ca-aa33-68e97550cc45 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:30.954316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959d70ae-2cbd-43ca-aa33-68e97550cc45 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:30.989834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d24f7ae-3a53-4b79-9ea1-63f071bce7a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:30.990843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d24f7ae-3a53-4b79-9ea1-63f071bce7a5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:30.991314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d24f7ae-3a53-4b79-9ea1-63f071bce7a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:30.991796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d24f7ae-3a53-4b79-9ea1-63f071bce7a5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:31.049832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:31.069824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d24f7ae-3a53-4b79-9ea1-63f071bce7a5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:31.071315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d24f7ae-3a53-4b79-9ea1-63f071bce7a5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:31.179952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 435/865] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:30 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:42:31.185994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68264cd5-5d7c-4cb0-9730-d1f8c665c2cf None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:31.215306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:31.298971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68264cd5-5d7c-4cb0-9730-d1f8c665c2cf None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:31.335349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545f73f5-cbb7-4c26-b779-aace237c554e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:31.336780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545f73f5-cbb7-4c26-b779-aace237c554e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:31.337249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545f73f5-cbb7-4c26-b779-aace237c554e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:31.337709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545f73f5-cbb7-4c26-b779-aace237c554e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:31.348119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545f73f5-cbb7-4c26-b779-aace237c554e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:31.349858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545f73f5-cbb7-4c26-b779-aace237c554e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-545f73f5-cbb7-4c26-b779-aace237c554e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:31.365381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:31.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:31.367292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:31.367292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:31.367292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:31.369331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 431/866] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:31.375042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d24f7ae-3a53-4b79-9ea1-63f071bce7a5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:31.389627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072f1c1-958a-4e51-9ff2-578eadc2c2a6 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:31.389912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072f1c1-958a-4e51-9ff2-578eadc2c2a6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:31.390223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072f1c1-958a-4e51-9ff2-578eadc2c2a6 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:31.391798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 432/867] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:31.397195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbb6456-7714-4436-9272-afcc2e4de8bc None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:31.397474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbb6456-7714-4436-9272-afcc2e4de8bc None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:31.397747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbb6456-7714-4436-9272-afcc2e4de8bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:31.416258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-2bbb6456-7714-4436-9272-afcc2e4de8bc None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:31.417452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2bbb6456-7714-4436-9272-afcc2e4de8bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:31.484251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:31.504404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 433/868] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:31 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:31.507039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d24f7ae-3a53-4b79-9ea1-63f071bce7a5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:31.542989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2ae73a-9bd9-4420-88dc-26bf508b8123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:31.543945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2ae73a-9bd9-4420-88dc-26bf508b8123 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:31.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2ae73a-9bd9-4420-88dc-26bf508b8123 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:31.544918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2ae73a-9bd9-4420-88dc-26bf508b8123 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:31.545741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2ae73a-9bd9-4420-88dc-26bf508b8123 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:31.546267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2ae73a-9bd9-4420-88dc-26bf508b8123 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:31.547909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2ae73a-9bd9-4420-88dc-26bf508b8123 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:31.548864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e42d8-460a-4c85-94a5-54b9bfa99258 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:31.549187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e42d8-460a-4c85-94a5-54b9bfa99258 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:31.549560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964e42d8-460a-4c85-94a5-54b9bfa99258 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:31.551658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 434/869] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:31.561520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 436/870] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:31.563105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b7fa4a-d7d9-4b4c-a1ff-cbc6cd4feaab None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:31.563461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b7fa4a-d7d9-4b4c-a1ff-cbc6cd4feaab None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:31.563829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b7fa4a-d7d9-4b4c-a1ff-cbc6cd4feaab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:31.564923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2ae73a-9bd9-4420-88dc-26bf508b8123 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:31.576851 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:31.587388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-72b7fa4a-d7d9-4b4c-a1ff-cbc6cd4feaab None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:31.588549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-72b7fa4a-d7d9-4b4c-a1ff-cbc6cd4feaab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:31.657660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 435/871] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:31 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:31.695170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2ae73a-9bd9-4420-88dc-26bf508b8123 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:31.728947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c59a2a-d7d3-4dfa-b356-788dc0547334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:31.729566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c59a2a-d7d3-4dfa-b356-788dc0547334 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:31.729841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c59a2a-d7d3-4dfa-b356-788dc0547334 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:31.730136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c59a2a-d7d3-4dfa-b356-788dc0547334 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:31.736857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c59a2a-d7d3-4dfa-b356-788dc0547334 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:31.738328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c59a2a-d7d3-4dfa-b356-788dc0547334 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a0c59a2a-d7d3-4dfa-b356-788dc0547334 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:31.749646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:31.750556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:31.751451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:31.751451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:31.751451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:31.751451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:31.751451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:31.751451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:31.751451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:31.751451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:31.751451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 437/872] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:31.758939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b7fa4a-d7d9-4b4c-a1ff-cbc6cd4feaab None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:31.792401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c59a2a-d7d3-4dfa-b356-788dc0547334 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:31.795182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:31.865107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b7fa4a-d7d9-4b4c-a1ff-cbc6cd4feaab None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:31.904974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f82cadc-33ea-4c47-b0f4-5ad654c96c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:31.905887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f82cadc-33ea-4c47-b0f4-5ad654c96c5b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:31.906627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f82cadc-33ea-4c47-b0f4-5ad654c96c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:31.907074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f82cadc-33ea-4c47-b0f4-5ad654c96c5b None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:31.908630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f82cadc-33ea-4c47-b0f4-5ad654c96c5b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:31.909109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f82cadc-33ea-4c47-b0f4-5ad654c96c5b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:31.910964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f82cadc-33ea-4c47-b0f4-5ad654c96c5b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:31.913604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c59a2a-d7d3-4dfa-b356-788dc0547334 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:31.914798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:31.933878 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 436/873] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:31.934825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f82cadc-33ea-4c47-b0f4-5ad654c96c5b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:31.966569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84763fbb-00e4-44f3-99fe-d1d2eb19f29b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:31.967593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84763fbb-00e4-44f3-99fe-d1d2eb19f29b None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:31.968000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84763fbb-00e4-44f3-99fe-d1d2eb19f29b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:31.968393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84763fbb-00e4-44f3-99fe-d1d2eb19f29b None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:31.969341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84763fbb-00e4-44f3-99fe-d1d2eb19f29b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:31.971114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84763fbb-00e4-44f3-99fe-d1d2eb19f29b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:31.998270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 438/874] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:42:31 2022] POST /identity/v3/services => generated 251 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:42:31.999250 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84763fbb-00e4-44f3-99fe-d1d2eb19f29b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:32.064963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f82cadc-33ea-4c47-b0f4-5ad654c96c5b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:32.098204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:32.130705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c262b94-ffac-40db-9079-46e491cd75f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:32.131864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c262b94-ffac-40db-9079-46e491cd75f9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:32.132550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c262b94-ffac-40db-9079-46e491cd75f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:32.133476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c262b94-ffac-40db-9079-46e491cd75f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:32.134716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c262b94-ffac-40db-9079-46e491cd75f9 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=db6c9dbbcf244a6f8bac59d2c9c973bc) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:32.135190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c262b94-ffac-40db-9079-46e491cd75f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:32.137310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c262b94-ffac-40db-9079-46e491cd75f9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:32.145528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 437/875] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:31 2022] GET /identity/v3/role_assignments?user.id=db6c9dbbcf244a6f8bac59d2c9c973bc&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 608 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:32.161865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84763fbb-00e4-44f3-99fe-d1d2eb19f29b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:32.201457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58f5e14-86f4-4a97-a41f-5585f7740fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:32.202452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58f5e14-86f4-4a97-a41f-5585f7740fba None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:32.202878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58f5e14-86f4-4a97-a41f-5585f7740fba None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:32.203302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58f5e14-86f4-4a97-a41f-5585f7740fba None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:32.204129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58f5e14-86f4-4a97-a41f-5585f7740fba None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:32.205697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58f5e14-86f4-4a97-a41f-5585f7740fba None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:32.225690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 439/876] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:42:31 2022] POST /identity/v3/services => generated 269 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:32.539010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ea5a1e-337c-478e-826b-54064057bf6f None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:32.539526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ea5a1e-337c-478e-826b-54064057bf6f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:32.539526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ea5a1e-337c-478e-826b-54064057bf6f None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:32.541193 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 438/877] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:32.546562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86800d00-0c40-4431-816b-87a263547b70 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:32.546848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86800d00-0c40-4431-816b-87a263547b70 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:32.547125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86800d00-0c40-4431-816b-87a263547b70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:32.565515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-86800d00-0c40-4431-816b-87a263547b70 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:32.566854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-86800d00-0c40-4431-816b-87a263547b70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:32.642511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 440/878] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:32 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:32.921587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ea5a1e-337c-478e-826b-54064057bf6f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:32.982362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:33.031762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ea5a1e-337c-478e-826b-54064057bf6f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:33.066837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd5ba75-1a5f-4ac4-9b4c-e41d2d349d3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:33.067979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd5ba75-1a5f-4ac4-9b4c-e41d2d349d3e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:33.068267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd5ba75-1a5f-4ac4-9b4c-e41d2d349d3e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:33.068537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd5ba75-1a5f-4ac4-9b4c-e41d2d349d3e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:33.079379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd5ba75-1a5f-4ac4-9b4c-e41d2d349d3e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-1dd5ba75-1a5f-4ac4-9b4c-e41d2d349d3e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:33.084314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:33.085303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:33.086246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 439/879] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 2 00:42:32 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:33.091545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86800d00-0c40-4431-816b-87a263547b70 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:33.144848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:33.204613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86800d00-0c40-4431-816b-87a263547b70 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:33.250842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9560cc-637d-4b40-b3c1-bf0afb557ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:33.251536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9560cc-637d-4b40-b3c1-bf0afb557ad5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:33.251815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9560cc-637d-4b40-b3c1-bf0afb557ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:33.252084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9560cc-637d-4b40-b3c1-bf0afb557ad5 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:33.252828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9560cc-637d-4b40-b3c1-bf0afb557ad5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:33.253133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9560cc-637d-4b40-b3c1-bf0afb557ad5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:33.256069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9560cc-637d-4b40-b3c1-bf0afb557ad5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:33.265556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 441/880] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 2 00:42:33 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:33.271771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd5ba75-1a5f-4ac4-9b4c-e41d2d349d3e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:33.390594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd5ba75-1a5f-4ac4-9b4c-e41d2d349d3e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:33.407504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:33.425834 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06afbc1-5891-4c0a-a3fc-3aa1fb0ecab2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:33.426490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06afbc1-5891-4c0a-a3fc-3aa1fb0ecab2 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:33.426790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06afbc1-5891-4c0a-a3fc-3aa1fb0ecab2 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:33.427082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06afbc1-5891-4c0a-a3fc-3aa1fb0ecab2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:33.437634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06afbc1-5891-4c0a-a3fc-3aa1fb0ecab2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:33.439088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06afbc1-5891-4c0a-a3fc-3aa1fb0ecab2 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-f06afbc1-5891-4c0a-a3fc-3aa1fb0ecab2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:33.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:33.454121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:33.455077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:33.455077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:33.455077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:33.455077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 440/881] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:33.460674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9560cc-637d-4b40-b3c1-bf0afb557ad5 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:33.561005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9560cc-637d-4b40-b3c1-bf0afb557ad5 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:33.575637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:33.600829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c9a3de-6db0-4168-ac59-a1a1c76161a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:33.601444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c9a3de-6db0-4168-ac59-a1a1c76161a4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:33.601720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c9a3de-6db0-4168-ac59-a1a1c76161a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:33.601991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c9a3de-6db0-4168-ac59-a1a1c76161a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:33.602655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c9a3de-6db0-4168-ac59-a1a1c76161a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:33.602952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c9a3de-6db0-4168-ac59-a1a1c76161a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:33.604315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c9a3de-6db0-4168-ac59-a1a1c76161a4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:33.617507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 442/882] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:33.622907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06afbc1-5891-4c0a-a3fc-3aa1fb0ecab2 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:33.733899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06afbc1-5891-4c0a-a3fc-3aa1fb0ecab2 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:33.768000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e2af2c-8845-4827-ab8d-5442b35905a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:33.768622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e2af2c-8845-4827-ab8d-5442b35905a4 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:33.768900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e2af2c-8845-4827-ab8d-5442b35905a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:33.769233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e2af2c-8845-4827-ab8d-5442b35905a4 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:33.776935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e2af2c-8845-4827-ab8d-5442b35905a4 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:33.778396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e2af2c-8845-4827-ab8d-5442b35905a4 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-07e2af2c-8845-4827-ab8d-5442b35905a4 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:33.790032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:33.791029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:33.792006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 441/883] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 2 00:42:33 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:33.799318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c9a3de-6db0-4168-ac59-a1a1c76161a4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:33.903596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c9a3de-6db0-4168-ac59-a1a1c76161a4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:33.938472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e7695f-00f6-4f77-954d-d3e09e1e616a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:33.939080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7695f-00f6-4f77-954d-d3e09e1e616a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:33.939363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7695f-00f6-4f77-954d-d3e09e1e616a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:33.939634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7695f-00f6-4f77-954d-d3e09e1e616a None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:33.940351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7695f-00f6-4f77-954d-d3e09e1e616a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:33.940651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7695f-00f6-4f77-954d-d3e09e1e616a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:33.942259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7695f-00f6-4f77-954d-d3e09e1e616a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:33.960815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 443/884] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Oct 2 00:42:33 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:33.967141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e2af2c-8845-4827-ab8d-5442b35905a4 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:33.996969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:34.075426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e2af2c-8845-4827-ab8d-5442b35905a4 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:34.110172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874675bf-a476-43df-96d2-96ca6699f7cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:34.110806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874675bf-a476-43df-96d2-96ca6699f7cc None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.111087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874675bf-a476-43df-96d2-96ca6699f7cc None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.111356 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874675bf-a476-43df-96d2-96ca6699f7cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.184831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874675bf-a476-43df-96d2-96ca6699f7cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:34.186313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874675bf-a476-43df-96d2-96ca6699f7cc None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:34.214231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:34.246018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ee747d-5a57-47e3-88b2-4fbdec110e5e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.246570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ee747d-5a57-47e3-88b2-4fbdec110e5e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.246570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ee747d-5a57-47e3-88b2-4fbdec110e5e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.248152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 444/885] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:34.253436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d335ff58-5728-40e9-993a-2acf73bbd667 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.253718 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d335ff58-5728-40e9-993a-2acf73bbd667 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.253984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d335ff58-5728-40e9-993a-2acf73bbd667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.272627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d335ff58-5728-40e9-993a-2acf73bbd667 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:34.273830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d335ff58-5728-40e9-993a-2acf73bbd667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:34.288276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 442/886] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:33 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:42:34.295053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874675bf-a476-43df-96d2-96ca6699f7cc None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:34.302838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:34.374050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 445/887] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:34 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:34.408154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874675bf-a476-43df-96d2-96ca6699f7cc None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:34.436122 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:34.444796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff3dd1-1747-48f4-b581-c0463792dc7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:34.445618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff3dd1-1747-48f4-b581-c0463792dc7e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.446040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff3dd1-1747-48f4-b581-c0463792dc7e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.446499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff3dd1-1747-48f4-b581-c0463792dc7e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.456850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff3dd1-1747-48f4-b581-c0463792dc7e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:34.458616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff3dd1-1747-48f4-b581-c0463792dc7e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:34.459587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ca7396-b36c-418f-ac3c-a9a1ba3ee51f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.459876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ca7396-b36c-418f-ac3c-a9a1ba3ee51f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.460188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ca7396-b36c-418f-ac3c-a9a1ba3ee51f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.461645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 446/888] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:34.466913 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f872c49-966f-40db-a8b8-46bf5006017c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.467202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f872c49-966f-40db-a8b8-46bf5006017c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.467468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f872c49-966f-40db-a8b8-46bf5006017c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-b7ff3dd1-1747-48f4-b581-c0463792dc7e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:34.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:34.474994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:34.475888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:34.475888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:34.475888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:34.475888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 443/889] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:34.481589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff3dd1-1747-48f4-b581-c0463792dc7e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:34.490655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2f872c49-966f-40db-a8b8-46bf5006017c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:34.492902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2f872c49-966f-40db-a8b8-46bf5006017c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:34.602186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 447/890] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:34 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:34.612995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff3dd1-1747-48f4-b581-c0463792dc7e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:34.621897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4641d4aa-0892-4087-8c1c-f6f7a72ea40c None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.622214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4641d4aa-0892-4087-8c1c-f6f7a72ea40c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.622470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4641d4aa-0892-4087-8c1c-f6f7a72ea40c None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.623938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 448/891] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:34.629116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128bf004-7510-4b30-bd60-6701776d286d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.629649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128bf004-7510-4b30-bd60-6701776d286d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.630700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128bf004-7510-4b30-bd60-6701776d286d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.650920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-128bf004-7510-4b30-bd60-6701776d286d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:34.652710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-128bf004-7510-4b30-bd60-6701776d286d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:34.656371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc0ff72-a917-490e-bbd3-ad0c97ee4046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:34.656975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc0ff72-a917-490e-bbd3-ad0c97ee4046 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.657248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc0ff72-a917-490e-bbd3-ad0c97ee4046 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.657517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc0ff72-a917-490e-bbd3-ad0c97ee4046 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.658177 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc0ff72-a917-490e-bbd3-ad0c97ee4046 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:34.658478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc0ff72-a917-490e-bbd3-ad0c97ee4046 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:34.659817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc0ff72-a917-490e-bbd3-ad0c97ee4046 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:34.673166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 444/892] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:34.673796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc0ff72-a917-490e-bbd3-ad0c97ee4046 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:34.723076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:34.727008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 449/893] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:34 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:34.731200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128bf004-7510-4b30-bd60-6701776d286d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:34.754358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:34.791613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc0ff72-a917-490e-bbd3-ad0c97ee4046 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:34.837957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb474320-7a17-430b-b671-f86bc1605610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:34.838609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb474320-7a17-430b-b671-f86bc1605610 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.838883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb474320-7a17-430b-b671-f86bc1605610 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.839151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb474320-7a17-430b-b671-f86bc1605610 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.839728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb474320-7a17-430b-b671-f86bc1605610 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:34.841187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb474320-7a17-430b-b671-f86bc1605610 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-bb474320-7a17-430b-b671-f86bc1605610 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:34.850302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:34.851348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:34.857206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 445/894] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:42:34 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:34.858027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb474320-7a17-430b-b671-f86bc1605610 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:34.866912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128bf004-7510-4b30-bd60-6701776d286d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:34.900622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e2b417-ba75-4ffb-99f2-368cbea884ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:34.901247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e2b417-ba75-4ffb-99f2-368cbea884ee None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:34.901518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e2b417-ba75-4ffb-99f2-368cbea884ee None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:34.901791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e2b417-ba75-4ffb-99f2-368cbea884ee None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:34.908471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e2b417-ba75-4ffb-99f2-368cbea884ee None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:34.909905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e2b417-ba75-4ffb-99f2-368cbea884ee None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-31e2b417-ba75-4ffb-99f2-368cbea884ee None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:34.921057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:34.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:34.922915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:34.922915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:34.922915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:34.922915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:34.922915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:34.922915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:34.922915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:34.922915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:34.927600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 450/895] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:34.928246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e2b417-ba75-4ffb-99f2-368cbea884ee None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:34.965909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb474320-7a17-430b-b671-f86bc1605610 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:34.969045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:35.002040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621cd112-fc18-48c5-b717-2622288af73e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.002857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621cd112-fc18-48c5-b717-2622288af73e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.003212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621cd112-fc18-48c5-b717-2622288af73e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.003686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621cd112-fc18-48c5-b717-2622288af73e None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.004669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621cd112-fc18-48c5-b717-2622288af73e None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:35.005179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621cd112-fc18-48c5-b717-2622288af73e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.006855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621cd112-fc18-48c5-b717-2622288af73e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.010612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:35.011783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 446/896] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:42:34 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:35.015227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621cd112-fc18-48c5-b717-2622288af73e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.034288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e2b417-ba75-4ffb-99f2-368cbea884ee None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.067061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b821f6e8-92c6-4628-b275-a1092427e8fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.068253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b821f6e8-92c6-4628-b275-a1092427e8fb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.068525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b821f6e8-92c6-4628-b275-a1092427e8fb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.068784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b821f6e8-92c6-4628-b275-a1092427e8fb None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.069347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b821f6e8-92c6-4628-b275-a1092427e8fb None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.070731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b821f6e8-92c6-4628-b275-a1092427e8fb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b821f6e8-92c6-4628-b275-a1092427e8fb None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:35.079486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:35.080525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:35.082030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 451/897] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:34 2022] GET /identity/v3/services/object-store => generated 93 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:35.082910 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b821f6e8-92c6-4628-b275-a1092427e8fb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.119909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621cd112-fc18-48c5-b717-2622288af73e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.155232 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3158ab02-843c-48e5-90e8-f2607f1b8fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.155832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3158ab02-843c-48e5-90e8-f2607f1b8fb5 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.156110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3158ab02-843c-48e5-90e8-f2607f1b8fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.156375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3158ab02-843c-48e5-90e8-f2607f1b8fb5 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.157053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3158ab02-843c-48e5-90e8-f2607f1b8fb5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:35.157352 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3158ab02-843c-48e5-90e8-f2607f1b8fb5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.158793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3158ab02-843c-48e5-90e8-f2607f1b8fb5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.170978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 447/898] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3158ab02-843c-48e5-90e8-f2607f1b8fb5 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.186305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b821f6e8-92c6-4628-b275-a1092427e8fb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.199038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:35.220342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b697a7-c431-40ea-ac34-554b5b1f0a56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.220967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b697a7-c431-40ea-ac34-554b5b1f0a56 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.221242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b697a7-c431-40ea-ac34-554b5b1f0a56 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.221510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b697a7-c431-40ea-ac34-554b5b1f0a56 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.222110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b697a7-c431-40ea-ac34-554b5b1f0a56 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.223467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b697a7-c431-40ea-ac34-554b5b1f0a56 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-00b697a7-c431-40ea-ac34-554b5b1f0a56 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:35.232645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:35.233611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:35.234563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 452/899] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:35.236008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b697a7-c431-40ea-ac34-554b5b1f0a56 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.243188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:35.283526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3158ab02-843c-48e5-90e8-f2607f1b8fb5 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.316737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e114ed5f-43b3-4b22-9c5b-c05d2473b879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.317880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114ed5f-43b3-4b22-9c5b-c05d2473b879 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.318189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114ed5f-43b3-4b22-9c5b-c05d2473b879 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.318467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114ed5f-43b3-4b22-9c5b-c05d2473b879 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.319097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e114ed5f-43b3-4b22-9c5b-c05d2473b879 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:35.319398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e114ed5f-43b3-4b22-9c5b-c05d2473b879 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.320763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e114ed5f-43b3-4b22-9c5b-c05d2473b879 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.325422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 448/900] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:35.326223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e114ed5f-43b3-4b22-9c5b-c05d2473b879 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.334441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b697a7-c431-40ea-ac34-554b5b1f0a56 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.367565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bb8d1c-1e78-495b-98df-e94d8d460c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.368166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb8d1c-1e78-495b-98df-e94d8d460c0a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.368441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb8d1c-1e78-495b-98df-e94d8d460c0a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.368718 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bb8d1c-1e78-495b-98df-e94d8d460c0a None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.369335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bb8d1c-1e78-495b-98df-e94d8d460c0a None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:35.369634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bb8d1c-1e78-495b-98df-e94d8d460c0a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.371071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bb8d1c-1e78-495b-98df-e94d8d460c0a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.375695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 453/901] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:35.376456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bb8d1c-1e78-495b-98df-e94d8d460c0a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.414817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:35.425749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e114ed5f-43b3-4b22-9c5b-c05d2473b879 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.447799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:35.461272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bd32a4-866a-4383-9f4b-05bfd7897236 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.461853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bd32a4-866a-4383-9f4b-05bfd7897236 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.462144 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bd32a4-866a-4383-9f4b-05bfd7897236 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.462411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bd32a4-866a-4383-9f4b-05bfd7897236 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.463115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bd32a4-866a-4383-9f4b-05bfd7897236 None admin] RBAC: Adding query filter params (role.id=3d250b53a014420ab6d096fcdfc1d9e1, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=af51bf378ea2491780250fc037fa2920) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:35.463415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bd32a4-866a-4383-9f4b-05bfd7897236 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.464807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bd32a4-866a-4383-9f4b-05bfd7897236 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.469851 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 449/902] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/role_assignments?user.id=af51bf378ea2491780250fc037fa2920&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=3d250b53a014420ab6d096fcdfc1d9e1 => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:35.470632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bd32a4-866a-4383-9f4b-05bfd7897236 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.479489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bb8d1c-1e78-495b-98df-e94d8d460c0a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.518871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14efd90c-13dd-4d4e-aae7-2376e962aab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.519494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14efd90c-13dd-4d4e-aae7-2376e962aab4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.519772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14efd90c-13dd-4d4e-aae7-2376e962aab4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.520047 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14efd90c-13dd-4d4e-aae7-2376e962aab4 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.521319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14efd90c-13dd-4d4e-aae7-2376e962aab4 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:35.521626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14efd90c-13dd-4d4e-aae7-2376e962aab4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.523255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14efd90c-13dd-4d4e-aae7-2376e962aab4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.529308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 454/903] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:35.530310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14efd90c-13dd-4d4e-aae7-2376e962aab4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.592010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bd32a4-866a-4383-9f4b-05bfd7897236 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.631038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cf37af-6117-4bfc-8ec4-444e734303c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.631569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cf37af-6117-4bfc-8ec4-444e734303c9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.632156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cf37af-6117-4bfc-8ec4-444e734303c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.632480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cf37af-6117-4bfc-8ec4-444e734303c9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.633184 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cf37af-6117-4bfc-8ec4-444e734303c9 None admin] RBAC: Adding query filter params (interface=public, service_id=fab48cbe369148dbaae0aea966f313ab, region_id=RegionOne) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:35.633494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cf37af-6117-4bfc-8ec4-444e734303c9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.634890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cf37af-6117-4bfc-8ec4-444e734303c9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.644266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 450/904] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/endpoints?service_id=fab48cbe369148dbaae0aea966f313ab&interface=public®ion_id=RegionOne => generated 194 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:35.644893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cf37af-6117-4bfc-8ec4-444e734303c9 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.651157 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14efd90c-13dd-4d4e-aae7-2376e962aab4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.673430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:35.687760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3e65a7-d8eb-4052-b14c-3936fb92c715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.688354 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3e65a7-d8eb-4052-b14c-3936fb92c715 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.688632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3e65a7-d8eb-4052-b14c-3936fb92c715 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.688904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3e65a7-d8eb-4052-b14c-3936fb92c715 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.689525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3e65a7-d8eb-4052-b14c-3936fb92c715 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:35.689824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3e65a7-d8eb-4052-b14c-3936fb92c715 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.691283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3e65a7-d8eb-4052-b14c-3936fb92c715 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.693214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:35.706091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 455/905] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:35.706091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3e65a7-d8eb-4052-b14c-3936fb92c715 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.787115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cf37af-6117-4bfc-8ec4-444e734303c9 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.817086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3e65a7-d8eb-4052-b14c-3936fb92c715 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:35.830777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e8f174-674c-483b-bbc0-3e413eea3ab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.831493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e8f174-674c-483b-bbc0-3e413eea3ab7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.831794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e8f174-674c-483b-bbc0-3e413eea3ab7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.832062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e8f174-674c-483b-bbc0-3e413eea3ab7 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.832690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e8f174-674c-483b-bbc0-3e413eea3ab7 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:35.833530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e8f174-674c-483b-bbc0-3e413eea3ab7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.834998 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e8f174-674c-483b-bbc0-3e413eea3ab7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.845512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 451/906] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:35.846525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e8f174-674c-483b-bbc0-3e413eea3ab7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:35.857955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cecb8a1-6d60-4900-9762-d9af38bf7ede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:35.858606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cecb8a1-6d60-4900-9762-d9af38bf7ede None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:35.858888 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cecb8a1-6d60-4900-9762-d9af38bf7ede None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:35.859271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cecb8a1-6d60-4900-9762-d9af38bf7ede None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:35.859894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cecb8a1-6d60-4900-9762-d9af38bf7ede None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:35.861416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cecb8a1-6d60-4900-9762-d9af38bf7ede None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:35.868894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 456/907] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/services => generated 1268 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:42:35.945564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:35.965795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e8f174-674c-483b-bbc0-3e413eea3ab7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:36.004156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c383640-8a67-4682-811b-80c23f03d55e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:36.005317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c383640-8a67-4682-811b-80c23f03d55e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:36.005605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c383640-8a67-4682-811b-80c23f03d55e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:36.005874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c383640-8a67-4682-811b-80c23f03d55e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:36.006626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c383640-8a67-4682-811b-80c23f03d55e None admin] RBAC: Adding query filter params (interface=public, service_id=1fa82b7c47a2412a8e19a1eb55cbe6a9, region_id=RegionOne) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:36.007237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c383640-8a67-4682-811b-80c23f03d55e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:36.008680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c383640-8a67-4682-811b-80c23f03d55e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:36.015274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 452/908] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 2 00:42:35 2022] GET /identity/v3/endpoints?service_id=1fa82b7c47a2412a8e19a1eb55cbe6a9&interface=public®ion_id=RegionOne => generated 194 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:36.023288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cecb8a1-6d60-4900-9762-d9af38bf7ede None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:36.094038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:36.135881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cecb8a1-6d60-4900-9762-d9af38bf7ede None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:36.171151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f032fe44-9584-40f5-bb29-bf51aae47d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:36.172168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f032fe44-9584-40f5-bb29-bf51aae47d94 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:36.172682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f032fe44-9584-40f5-bb29-bf51aae47d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:36.173173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f032fe44-9584-40f5-bb29-bf51aae47d94 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:36.174101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f032fe44-9584-40f5-bb29-bf51aae47d94 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:36.175653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f032fe44-9584-40f5-bb29-bf51aae47d94 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:36.182087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 457/909] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 2 00:42:36 2022] GET /identity/v3/services => generated 1268 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:42:37.657893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3df2bc-39ec-489f-a6af-3796d9ccc953 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:37.658460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3df2bc-39ec-489f-a6af-3796d9ccc953 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:37.658460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3df2bc-39ec-489f-a6af-3796d9ccc953 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:37.660117 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 453/910] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:37.665903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a0cbd-cd8d-46f7-aeff-fe3415349af8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:37.666379 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a0cbd-cd8d-46f7-aeff-fe3415349af8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:37.666477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a0cbd-cd8d-46f7-aeff-fe3415349af8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:37.684915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-616a0cbd-cd8d-46f7-aeff-fe3415349af8 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:37.686112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-616a0cbd-cd8d-46f7-aeff-fe3415349af8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:37.755278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 458/911] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:37 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:37.949664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d62aa-c2b6-4f82-abca-aae2d965c02b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:37.949863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d62aa-c2b6-4f82-abca-aae2d965c02b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:37.950151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d62aa-c2b6-4f82-abca-aae2d965c02b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:37.951690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 454/912] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:37.957032 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c51e9b1-9c3b-4cbd-bb36-96f0d2fb8db3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:37.957313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c51e9b1-9c3b-4cbd-bb36-96f0d2fb8db3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:37.957583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c51e9b1-9c3b-4cbd-bb36-96f0d2fb8db3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:37.976664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5c51e9b1-9c3b-4cbd-bb36-96f0d2fb8db3 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:37.977857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5c51e9b1-9c3b-4cbd-bb36-96f0d2fb8db3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:38.028743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d62aa-c2b6-4f82-abca-aae2d965c02b None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:38.052733 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:38.074819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 459/913] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:37 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:38.120677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a7ff4b-cad2-4005-aa9a-380bf72ceb49 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.121313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a7ff4b-cad2-4005-aa9a-380bf72ceb49 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.121820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a7ff4b-cad2-4005-aa9a-380bf72ceb49 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.123713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 460/914] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:38.128857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb465293-f6ca-4880-b763-24cd0ec97720 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.129379 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb465293-f6ca-4880-b763-24cd0ec97720 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.129943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb465293-f6ca-4880-b763-24cd0ec97720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.150603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d62aa-c2b6-4f82-abca-aae2d965c02b None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:38.151357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cb465293-f6ca-4880-b763-24cd0ec97720 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:38.153648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cb465293-f6ca-4880-b763-24cd0ec97720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:38.191514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5adeab-2714-4a9b-9847-7fee07548d11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:38.192225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5adeab-2714-4a9b-9847-7fee07548d11 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.192513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5adeab-2714-4a9b-9847-7fee07548d11 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.192790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5adeab-2714-4a9b-9847-7fee07548d11 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.207869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5adeab-2714-4a9b-9847-7fee07548d11 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-0d5adeab-2714-4a9b-9847-7fee07548d11 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:38.213627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:38.214641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:38.219152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 455/915] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 2 00:42:38 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:38.221804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5adeab-2714-4a9b-9847-7fee07548d11 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:38.234672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:38.253598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 461/916] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:38 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:38.339950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5adeab-2714-4a9b-9847-7fee07548d11 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:38.347577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb465293-f6ca-4880-b763-24cd0ec97720 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:38.382853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f064f9-a7b7-4a5b-aa0e-a91f19c70347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:38.383467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f064f9-a7b7-4a5b-aa0e-a91f19c70347 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.384560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f064f9-a7b7-4a5b-aa0e-a91f19c70347 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.384560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f064f9-a7b7-4a5b-aa0e-a91f19c70347 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.384732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f064f9-a7b7-4a5b-aa0e-a91f19c70347 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:38.384997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f064f9-a7b7-4a5b-aa0e-a91f19c70347 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:38.386370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f064f9-a7b7-4a5b-aa0e-a91f19c70347 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:38.398123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 456/917] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 2 00:42:38 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:38.398123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504fe680-51c9-47fc-8acc-ede71ac76642 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.398123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504fe680-51c9-47fc-8acc-ede71ac76642 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.398123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504fe680-51c9-47fc-8acc-ede71ac76642 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.398123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 457/918] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:38.402625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504fe680-51c9-47fc-8acc-ede71ac76642 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:38.454330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb465293-f6ca-4880-b763-24cd0ec97720 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:38.471284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:38.478913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:38.490339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9cc1ee-be74-4bc1-a045-d32667030c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:38.490952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9cc1ee-be74-4bc1-a045-d32667030c27 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.491235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9cc1ee-be74-4bc1-a045-d32667030c27 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.491494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9cc1ee-be74-4bc1-a045-d32667030c27 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.492132 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9cc1ee-be74-4bc1-a045-d32667030c27 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:38.493494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a9cc1ee-be74-4bc1-a045-d32667030c27 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:38.508096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 462/919] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:42:38 2022] POST /identity/v3/services => generated 241 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:42:38.515797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352915c7-7342-4c5c-bdd9-9b6117bd74e7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.516228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352915c7-7342-4c5c-bdd9-9b6117bd74e7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.516649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352915c7-7342-4c5c-bdd9-9b6117bd74e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.523926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504fe680-51c9-47fc-8acc-ede71ac76642 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:38.543199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-352915c7-7342-4c5c-bdd9-9b6117bd74e7 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:38.544995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-352915c7-7342-4c5c-bdd9-9b6117bd74e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:38.566228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09d8940-cf54-49ae-b4ff-2b8c3c2a6c85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:38.566876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09d8940-cf54-49ae-b4ff-2b8c3c2a6c85 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.567192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09d8940-cf54-49ae-b4ff-2b8c3c2a6c85 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.567460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09d8940-cf54-49ae-b4ff-2b8c3c2a6c85 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.579046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09d8940-cf54-49ae-b4ff-2b8c3c2a6c85 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:38.580521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09d8940-cf54-49ae-b4ff-2b8c3c2a6c85 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-f09d8940-cf54-49ae-b4ff-2b8c3c2a6c85 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:38.596215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:38.597492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:38.598797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:38.598797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:38.598797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:38.602421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 458/920] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:38.603399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09d8940-cf54-49ae-b4ff-2b8c3c2a6c85 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:38.634089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 463/921] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:38 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:38.634089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352915c7-7342-4c5c-bdd9-9b6117bd74e7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:38.735408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09d8940-cf54-49ae-b4ff-2b8c3c2a6c85 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:38.760437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:38.777576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352915c7-7342-4c5c-bdd9-9b6117bd74e7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:38.782443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e162570-ecd4-4ad9-b637-fca46afd95ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:38.783071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e162570-ecd4-4ad9-b637-fca46afd95ec None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.783340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e162570-ecd4-4ad9-b637-fca46afd95ec None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.783606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e162570-ecd4-4ad9-b637-fca46afd95ec None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.784181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e162570-ecd4-4ad9-b637-fca46afd95ec None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:38.785565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:38.787142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e162570-ecd4-4ad9-b637-fca46afd95ec None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-0e162570-ecd4-4ad9-b637-fca46afd95ec None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:38.800401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:38.801669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:38.803022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 459/922] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:42:38 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:38.807422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e162570-ecd4-4ad9-b637-fca46afd95ec None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:38.822256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37584e41-0f06-47d8-b585-2c8bf750be65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:38.823191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37584e41-0f06-47d8-b585-2c8bf750be65 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.823678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37584e41-0f06-47d8-b585-2c8bf750be65 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.824149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37584e41-0f06-47d8-b585-2c8bf750be65 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.825045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37584e41-0f06-47d8-b585-2c8bf750be65 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:38.825564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37584e41-0f06-47d8-b585-2c8bf750be65 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:38.827219 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37584e41-0f06-47d8-b585-2c8bf750be65 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:38.840091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 464/923] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:38.846315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37584e41-0f06-47d8-b585-2c8bf750be65 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:38.928006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e162570-ecd4-4ad9-b637-fca46afd95ec None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:38.953017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37584e41-0f06-47d8-b585-2c8bf750be65 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:38.962139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821be070-e39d-4794-9d59-51c33ea0b455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:38.962740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821be070-e39d-4794-9d59-51c33ea0b455 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.963012 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821be070-e39d-4794-9d59-51c33ea0b455 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.963278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821be070-e39d-4794-9d59-51c33ea0b455 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:38.963902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821be070-e39d-4794-9d59-51c33ea0b455 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:38.964197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821be070-e39d-4794-9d59-51c33ea0b455 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:38.965532 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821be070-e39d-4794-9d59-51c33ea0b455 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:38.971258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 460/924] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:42:38 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:38.971258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821be070-e39d-4794-9d59-51c33ea0b455 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:38.993922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fbc696-9bd2-4c72-acfb-7d1877920ed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:38.994828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fbc696-9bd2-4c72-acfb-7d1877920ed3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:38.995110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fbc696-9bd2-4c72-acfb-7d1877920ed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:38.995386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fbc696-9bd2-4c72-acfb-7d1877920ed3 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.003021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fbc696-9bd2-4c72-acfb-7d1877920ed3 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.004485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fbc696-9bd2-4c72-acfb-7d1877920ed3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e5fbc696-9bd2-4c72-acfb-7d1877920ed3 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:39.016480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:39.017596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 465/925] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 2 00:42:38 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:39.018575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fbc696-9bd2-4c72-acfb-7d1877920ed3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:39.030826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:39.058485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:39.091775 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821be070-e39d-4794-9d59-51c33ea0b455 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:39.128445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528f71ae-7fcd-4464-ae1c-a16f2b7d296d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:39.129070 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528f71ae-7fcd-4464-ae1c-a16f2b7d296d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:39.129346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528f71ae-7fcd-4464-ae1c-a16f2b7d296d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:39.129645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528f71ae-7fcd-4464-ae1c-a16f2b7d296d None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.130252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528f71ae-7fcd-4464-ae1c-a16f2b7d296d None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.131604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528f71ae-7fcd-4464-ae1c-a16f2b7d296d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.133932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fbc696-9bd2-4c72-acfb-7d1877920ed3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-528f71ae-7fcd-4464-ae1c-a16f2b7d296d None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:39.141946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:39.142915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:39.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 461/926] 10.222.0.25 () {56 vars in 1132 bytes} [Sun Oct 2 00:42:38 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:39.145115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528f71ae-7fcd-4464-ae1c-a16f2b7d296d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:39.173033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d36d6-3232-47ae-a4a2-cfb1811bab22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:39.173634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d36d6-3232-47ae-a4a2-cfb1811bab22 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:39.173914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d36d6-3232-47ae-a4a2-cfb1811bab22 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:39.174236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d36d6-3232-47ae-a4a2-cfb1811bab22 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.174859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d36d6-3232-47ae-a4a2-cfb1811bab22 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:39.175170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d36d6-3232-47ae-a4a2-cfb1811bab22 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.176661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d36d6-3232-47ae-a4a2-cfb1811bab22 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.181402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 466/927] 10.222.0.25 () {56 vars in 1114 bytes} [Sun Oct 2 00:42:39 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:39.182296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d36d6-3232-47ae-a4a2-cfb1811bab22 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:39.257620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528f71ae-7fcd-4464-ae1c-a16f2b7d296d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:39.268956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:39.287662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:39.297420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4e6ae0-3b47-49f2-8922-8f1147c0b750 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:39.298027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4e6ae0-3b47-49f2-8922-8f1147c0b750 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:39.298170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d36d6-3232-47ae-a4a2-cfb1811bab22 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:39.298336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4e6ae0-3b47-49f2-8922-8f1147c0b750 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:39.298599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4e6ae0-3b47-49f2-8922-8f1147c0b750 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.299329 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4e6ae0-3b47-49f2-8922-8f1147c0b750 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:39.299751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4e6ae0-3b47-49f2-8922-8f1147c0b750 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.301289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4e6ae0-3b47-49f2-8922-8f1147c0b750 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.326403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 462/928] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Oct 2 00:42:39 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:39.327631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4e6ae0-3b47-49f2-8922-8f1147c0b750 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:39.334694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6250a9-667e-4651-bd20-f9c7fd3169bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:39.335297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6250a9-667e-4651-bd20-f9c7fd3169bc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:39.335567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6250a9-667e-4651-bd20-f9c7fd3169bc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:39.335832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6250a9-667e-4651-bd20-f9c7fd3169bc None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.336450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6250a9-667e-4651-bd20-f9c7fd3169bc None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:39.337154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6250a9-667e-4651-bd20-f9c7fd3169bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.338549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6250a9-667e-4651-bd20-f9c7fd3169bc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.346218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 467/929] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:39 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:39.347289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6250a9-667e-4651-bd20-f9c7fd3169bc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:39.435967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4e6ae0-3b47-49f2-8922-8f1147c0b750 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:39.451590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6250a9-667e-4651-bd20-f9c7fd3169bc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:39.477849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7a22b5-dfc7-49a9-9df1-a0ec2481c658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:39.478507 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7a22b5-dfc7-49a9-9df1-a0ec2481c658 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:39.478782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7a22b5-dfc7-49a9-9df1-a0ec2481c658 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:39.479192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7a22b5-dfc7-49a9-9df1-a0ec2481c658 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.479705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7a22b5-dfc7-49a9-9df1-a0ec2481c658 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.481045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac7a22b5-dfc7-49a9-9df1-a0ec2481c658 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.488406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356f37e-fe82-4e4e-9cf2-800e98aa6354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:39.489661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356f37e-fe82-4e4e-9cf2-800e98aa6354 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:39.490008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356f37e-fe82-4e4e-9cf2-800e98aa6354 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:39.490374 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2356f37e-fe82-4e4e-9cf2-800e98aa6354 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.510182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 463/930] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:42:39 2022] POST /identity/v3/endpoints => generated 346 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:39.510182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7a22b5-dfc7-49a9-9df1-a0ec2481c658 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:39.510471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2356f37e-fe82-4e4e-9cf2-800e98aa6354 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.510471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2356f37e-fe82-4e4e-9cf2-800e98aa6354 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2356f37e-fe82-4e4e-9cf2-800e98aa6354 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:39.527085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:39.528229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:39.529121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:39.529121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:39.529121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:39.533467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 468/931] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:39.534886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356f37e-fe82-4e4e-9cf2-800e98aa6354 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:39.542712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:39.545295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:39.636235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7a22b5-dfc7-49a9-9df1-a0ec2481c658 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:39.648705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2356f37e-fe82-4e4e-9cf2-800e98aa6354 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:39.678989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae6a835-1e39-473b-add5-dccac51c01ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:39.679612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae6a835-1e39-473b-add5-dccac51c01ba None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:39.679888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae6a835-1e39-473b-add5-dccac51c01ba None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:39.680157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae6a835-1e39-473b-add5-dccac51c01ba None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.680795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae6a835-1e39-473b-add5-dccac51c01ba None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:39.681115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae6a835-1e39-473b-add5-dccac51c01ba None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.682723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae6a835-1e39-473b-add5-dccac51c01ba None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.690224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 464/932] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:39 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:39.695969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae6a835-1e39-473b-add5-dccac51c01ba None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:39.699529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d4177a-edf4-43ce-b32a-d2f51bdcd601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:39.700174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d4177a-edf4-43ce-b32a-d2f51bdcd601 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:39.700431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d4177a-edf4-43ce-b32a-d2f51bdcd601 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:39.700705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d4177a-edf4-43ce-b32a-d2f51bdcd601 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.702045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d4177a-edf4-43ce-b32a-d2f51bdcd601 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:39.702045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d4177a-edf4-43ce-b32a-d2f51bdcd601 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.703167 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d4177a-edf4-43ce-b32a-d2f51bdcd601 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.716989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 469/933] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:39.722537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d4177a-edf4-43ce-b32a-d2f51bdcd601 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:39.799638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:39.802623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:39.828581 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae6a835-1e39-473b-add5-dccac51c01ba None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:39.845276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d4177a-edf4-43ce-b32a-d2f51bdcd601 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:39.873251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10d927a-b007-4f08-aae4-42b402b8e5a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:39.873858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d927a-b007-4f08-aae4-42b402b8e5a4 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:39.874231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d927a-b007-4f08-aae4-42b402b8e5a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:39.874505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d927a-b007-4f08-aae4-42b402b8e5a4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.875265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10d927a-b007-4f08-aae4-42b402b8e5a4 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.876575 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10d927a-b007-4f08-aae4-42b402b8e5a4 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.887987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91777262-eef5-494e-b398-1a73ad0b68ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:39.889069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91777262-eef5-494e-b398-1a73ad0b68ff None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:39.889321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91777262-eef5-494e-b398-1a73ad0b68ff None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:39.889706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91777262-eef5-494e-b398-1a73ad0b68ff None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:39.902935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 465/934] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:42:39 2022] POST /identity/v3/endpoints => generated 347 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:39.904554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91777262-eef5-494e-b398-1a73ad0b68ff None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:39.906034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91777262-eef5-494e-b398-1a73ad0b68ff None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-91777262-eef5-494e-b398-1a73ad0b68ff None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:39.924531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:39.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:39.926535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:39.926535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:39.926535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:39.926535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:39.926535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:39.926535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:39.926535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:39.926535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:39.930252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 470/935] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:39.932322 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10d927a-b007-4f08-aae4-42b402b8e5a4 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:40.057019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10d927a-b007-4f08-aae4-42b402b8e5a4 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:40.093711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc564db-3b87-46c8-ae5d-358e1eebee81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:40.094362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc564db-3b87-46c8-ae5d-358e1eebee81 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:40.094638 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc564db-3b87-46c8-ae5d-358e1eebee81 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:40.094908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc564db-3b87-46c8-ae5d-358e1eebee81 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:40.095670 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc564db-3b87-46c8-ae5d-358e1eebee81 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:40.096036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc564db-3b87-46c8-ae5d-358e1eebee81 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:40.097464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc564db-3b87-46c8-ae5d-358e1eebee81 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:40.100192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:40.113536 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 466/936] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:40.118450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91777262-eef5-494e-b398-1a73ad0b68ff None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:40.225987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91777262-eef5-494e-b398-1a73ad0b68ff None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:40.253109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:40.263180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755e24e2-8113-44d2-b737-cbfb6e7edb70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:40.264134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755e24e2-8113-44d2-b737-cbfb6e7edb70 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:40.264648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755e24e2-8113-44d2-b737-cbfb6e7edb70 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:40.265113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755e24e2-8113-44d2-b737-cbfb6e7edb70 None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/af51bf378ea2491780250fc037fa2920/roles/3d250b53a014420ab6d096fcdfc1d9e1` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:40.267657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755e24e2-8113-44d2-b737-cbfb6e7edb70 None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=af51bf378ea2491780250fc037fa2920, role_id=3d250b53a014420ab6d096fcdfc1d9e1)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:40.269306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755e24e2-8113-44d2-b737-cbfb6e7edb70 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:40.285690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 471/937] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:42:40 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/af51bf378ea2491780250fc037fa2920/roles/3d250b53a014420ab6d096fcdfc1d9e1 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:42:40.758558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a049cd-4888-4f6f-9b10-43d27dc18316 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:40.758955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a049cd-4888-4f6f-9b10-43d27dc18316 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:40.759155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a049cd-4888-4f6f-9b10-43d27dc18316 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:40.760678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 467/938] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:40.765958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a64e77-b3b3-4ffc-938d-2b808ebb0ef2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:40.766267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a64e77-b3b3-4ffc-938d-2b808ebb0ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:40.766535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a64e77-b3b3-4ffc-938d-2b808ebb0ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:40.786566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c3a64e77-b3b3-4ffc-938d-2b808ebb0ef2 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:40.787747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c3a64e77-b3b3-4ffc-938d-2b808ebb0ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:40.854769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 472/939] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:40 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:41.127714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a049cd-4888-4f6f-9b10-43d27dc18316 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:41.248378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a049cd-4888-4f6f-9b10-43d27dc18316 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:41.283566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8880bd3d-13a2-4d79-aed9-7f5a05b33fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:41.284125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8880bd3d-13a2-4d79-aed9-7f5a05b33fb7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:41.284402 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8880bd3d-13a2-4d79-aed9-7f5a05b33fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:41.284675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8880bd3d-13a2-4d79-aed9-7f5a05b33fb7 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:41.285253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8880bd3d-13a2-4d79-aed9-7f5a05b33fb7 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:41.287179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8880bd3d-13a2-4d79-aed9-7f5a05b33fb7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-8880bd3d-13a2-4d79-aed9-7f5a05b33fb7 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:41.296009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:41.297733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:41.298672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 468/940] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:41 2022] GET /identity/v3/services/object-store => generated 93 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:41.303517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a64e77-b3b3-4ffc-938d-2b808ebb0ef2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:41.409502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a64e77-b3b3-4ffc-938d-2b808ebb0ef2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:41.427297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:41.451207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8aadb8-f98c-4440-8884-544a82c850cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:41.451814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8aadb8-f98c-4440-8884-544a82c850cf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:41.452088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8aadb8-f98c-4440-8884-544a82c850cf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:41.452357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8aadb8-f98c-4440-8884-544a82c850cf None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:41.452982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8aadb8-f98c-4440-8884-544a82c850cf None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:41.453283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8aadb8-f98c-4440-8884-544a82c850cf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:41.454694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8aadb8-f98c-4440-8884-544a82c850cf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:41.460122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 473/941] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 2 00:42:41 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:41.467134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8880bd3d-13a2-4d79-aed9-7f5a05b33fb7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:41.531326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:41.574324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8880bd3d-13a2-4d79-aed9-7f5a05b33fb7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:41.608426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191d99be-923f-49ef-bbc0-49f8bfa8cc40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:41.609025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191d99be-923f-49ef-bbc0-49f8bfa8cc40 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:41.609299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191d99be-923f-49ef-bbc0-49f8bfa8cc40 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:41.609614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191d99be-923f-49ef-bbc0-49f8bfa8cc40 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:41.610267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191d99be-923f-49ef-bbc0-49f8bfa8cc40 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:41.610566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191d99be-923f-49ef-bbc0-49f8bfa8cc40 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:41.611946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191d99be-923f-49ef-bbc0-49f8bfa8cc40 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:41.617685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 469/942] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 2 00:42:41 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:41.623644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8aadb8-f98c-4440-8884-544a82c850cf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:41.735109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8aadb8-f98c-4440-8884-544a82c850cf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:41.774247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1fe13c-e449-453c-9d75-8f62c3e137e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:41.774857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1fe13c-e449-453c-9d75-8f62c3e137e7 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:41.775133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1fe13c-e449-453c-9d75-8f62c3e137e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:41.775403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1fe13c-e449-453c-9d75-8f62c3e137e7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:41.776098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1fe13c-e449-453c-9d75-8f62c3e137e7 None admin] RBAC: Adding query filter params (interface=public, service_id=7732602dec424afdadd3388536f52caa, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:41.776401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1fe13c-e449-453c-9d75-8f62c3e137e7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:41.777762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1fe13c-e449-453c-9d75-8f62c3e137e7 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:41.783299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 474/943] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 2 00:42:41 2022] GET /identity/v3/endpoints?service_id=7732602dec424afdadd3388536f52caa&interface=public®ion_id=RegionOne => generated 194 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:41.788989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191d99be-923f-49ef-bbc0-49f8bfa8cc40 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:41.828079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563e5095-cc36-49f2-856c-a92a51c9caf2 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:41.828362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563e5095-cc36-49f2-856c-a92a51c9caf2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:41.828575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563e5095-cc36-49f2-856c-a92a51c9caf2 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:41.830150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 475/944] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:41.839360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f665b3-6b0d-4ef1-a9af-b3ed694ae42a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:41.839905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f665b3-6b0d-4ef1-a9af-b3ed694ae42a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:41.840269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f665b3-6b0d-4ef1-a9af-b3ed694ae42a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:41.858986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e5f665b3-6b0d-4ef1-a9af-b3ed694ae42a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:41.860299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e5f665b3-6b0d-4ef1-a9af-b3ed694ae42a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:41.902228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191d99be-923f-49ef-bbc0-49f8bfa8cc40 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:41.909919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:41.930574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:41.943516 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 476/945] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:41 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:41.951543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef72624-ef95-4e23-9c7a-79611bd18418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:41.952392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef72624-ef95-4e23-9c7a-79611bd18418 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:41.952871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef72624-ef95-4e23-9c7a-79611bd18418 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:41.953333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef72624-ef95-4e23-9c7a-79611bd18418 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:41.954159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef72624-ef95-4e23-9c7a-79611bd18418 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:41.955712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ef72624-ef95-4e23-9c7a-79611bd18418 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:41.962450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 470/946] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 2 00:42:41 2022] GET /identity/v3/services => generated 1497 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:42:42.221475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f665b3-6b0d-4ef1-a9af-b3ed694ae42a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:42.228593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3169b6c4-a402-4edc-9ce2-67dc653573f2 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:42.229076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3169b6c4-a402-4edc-9ce2-67dc653573f2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:42.229155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3169b6c4-a402-4edc-9ce2-67dc653573f2 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:42.230762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 471/947] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:42.235847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656c2741-4e78-40d2-81ac-a21c8421cd03 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:42.236126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656c2741-4e78-40d2-81ac-a21c8421cd03 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:42.236396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656c2741-4e78-40d2-81ac-a21c8421cd03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:42.257180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-656c2741-4e78-40d2-81ac-a21c8421cd03 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:42.258410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-656c2741-4e78-40d2-81ac-a21c8421cd03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:42.333798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 472/948] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:42.344846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f665b3-6b0d-4ef1-a9af-b3ed694ae42a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:42.380701 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:42.381666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:42.382429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:42.382896 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:42.383783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:42.385477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:42.388299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: WARNING keystone.server.flask.common [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 02 00:42:42.406782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.sql.core [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 02 00:42:42.406782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 02 00:42:42.406782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [parameters: {'id': 'ce6b1b8b828a44bcb096fd1306f95281', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 02 00:42:42.406782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81433) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ce6b1b8b828a44bcb096fd1306f95281.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ce6b1b8b828a44bcb096fd1306f95281. Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = self._query(query) Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application conn.query(q) Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 02 00:42:42.415341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result.read() Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 02 00:42:42.416159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application next(self.gen) Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application yield resource Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application next(self.gen) Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.session.rollback() Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.force_reraise() Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise self.value Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application session.commit() Oct 02 00:42:42.417115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._prepare_impl() Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.session.flush() Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._flush(objects) Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application compat.raise_( Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application flush_context.execute() Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 02 00:42:42.418034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rec.execute(self) Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 02 00:42:42.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = self._query(query) Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application conn.query(q) Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result.read() Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 02 00:42:42.419768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [parameters: {'id': 'ce6b1b8b828a44bcb096fd1306f95281', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:42.420632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:42.421424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:42.424043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:42.424043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 02 00:42:42.424043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 02 00:42:42.424043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:42.424043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:42.424043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 02 00:42:42.424043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 02 00:42:42.424043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ce6b1b8b828a44bcb096fd1306f95281. Oct 02 00:42:42.424043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:42.427412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 477/949] 10.222.0.25 () {60 vars in 1140 bytes} [Sun Oct 2 00:42:42 2022] POST /identity/v3/projects => generated 262 bytes in 206 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:42.434164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656c2741-4e78-40d2-81ac-a21c8421cd03 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:42.548373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656c2741-4e78-40d2-81ac-a21c8421cd03 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:42.567232 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:42.585434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88611b71-f8b3-400e-9dda-7d4cd2b3a81d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:42.586086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88611b71-f8b3-400e-9dda-7d4cd2b3a81d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:42.586372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88611b71-f8b3-400e-9dda-7d4cd2b3a81d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:42.586643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88611b71-f8b3-400e-9dda-7d4cd2b3a81d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:42.593348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88611b71-f8b3-400e-9dda-7d4cd2b3a81d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:42.594915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88611b71-f8b3-400e-9dda-7d4cd2b3a81d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-88611b71-f8b3-400e-9dda-7d4cd2b3a81d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:42.606360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:42.607246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:42.608132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:42.608132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:42.608132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:42.608132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:42.608132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:42.608132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:42.608132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:42.608132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:42.611769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 473/950] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:42.613043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:42.613523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88611b71-f8b3-400e-9dda-7d4cd2b3a81d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:42.644685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:42.719685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95544ae3-ba70-46a0-b2dd-f10b731b249a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:42.720297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88611b71-f8b3-400e-9dda-7d4cd2b3a81d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:42.753262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca128fb-241c-4934-b321-19c9b2147d57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:42.753843 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b3c9d4-0b3b-40f8-854b-8c78c5789bbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:42.753961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca128fb-241c-4934-b321-19c9b2147d57 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:42.754196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca128fb-241c-4934-b321-19c9b2147d57 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:42.754995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca128fb-241c-4934-b321-19c9b2147d57 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:42.755091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b3c9d4-0b3b-40f8-854b-8c78c5789bbc None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:42.755272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b3c9d4-0b3b-40f8-854b-8c78c5789bbc None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:42.755543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b3c9d4-0b3b-40f8-854b-8c78c5789bbc None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:42.755635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca128fb-241c-4934-b321-19c9b2147d57 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:42.756201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b3c9d4-0b3b-40f8-854b-8c78c5789bbc None admin] RBAC: Adding query filter params (name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:42.756504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b3c9d4-0b3b-40f8-854b-8c78c5789bbc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:42.757386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca128fb-241c-4934-b321-19c9b2147d57 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:42.758209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b3c9d4-0b3b-40f8-854b-8c78c5789bbc None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bca128fb-241c-4934-b321-19c9b2147d57 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:42.766327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:42.767318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 02 00:42:42.768339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:42.772389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 478/951] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:42 2022] GET /identity/v3/services/compute => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:42.773760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44976a6a-d0d3-4e9a-ac35-a317b6e7ab23 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:42.774041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44976a6a-d0d3-4e9a-ac35-a317b6e7ab23 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:42.774334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44976a6a-d0d3-4e9a-ac35-a317b6e7ab23 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:42.786396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 474/952] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:42 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:42.786613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 479/953] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:42 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:42.786836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b3c9d4-0b3b-40f8-854b-8c78c5789bbc None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:42.786937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376bcbce-83ea-47f8-a156-b88992bede29 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:42.786937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376bcbce-83ea-47f8-a156-b88992bede29 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:42.786937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376bcbce-83ea-47f8-a156-b88992bede29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:42.808459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-376bcbce-83ea-47f8-a156-b88992bede29 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:42.810616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-376bcbce-83ea-47f8-a156-b88992bede29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:42.831666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:42.900399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b3c9d4-0b3b-40f8-854b-8c78c5789bbc None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:42.914557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 480/954] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:42.949373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e471fd-a6fc-490a-9334-8677a0e3c2ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:42.949997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e471fd-a6fc-490a-9334-8677a0e3c2ca None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:42.950302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e471fd-a6fc-490a-9334-8677a0e3c2ca None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:42.950572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e471fd-a6fc-490a-9334-8677a0e3c2ca None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:42.951203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e471fd-a6fc-490a-9334-8677a0e3c2ca None admin] RBAC: Adding query filter params (name=compute) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:42.951499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e471fd-a6fc-490a-9334-8677a0e3c2ca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:42.952943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e471fd-a6fc-490a-9334-8677a0e3c2ca None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:42.959375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 475/955] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:42 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:42.965159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376bcbce-83ea-47f8-a156-b88992bede29 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:42.991551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:43.070917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376bcbce-83ea-47f8-a156-b88992bede29 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:43.107050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876b8bc-ab97-435a-8e5d-8030f727d46d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:43.108015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876b8bc-ab97-435a-8e5d-8030f727d46d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:43.108498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876b8bc-ab97-435a-8e5d-8030f727d46d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:43.108999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876b8bc-ab97-435a-8e5d-8030f727d46d None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:43.109838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f876b8bc-ab97-435a-8e5d-8030f727d46d None admin] RBAC: Adding query filter params (type=compute) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:43.110421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f876b8bc-ab97-435a-8e5d-8030f727d46d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:43.112076 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f876b8bc-ab97-435a-8e5d-8030f727d46d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:43.117897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 481/956] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:42 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:43.205158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e471fd-a6fc-490a-9334-8677a0e3c2ca None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:43.314835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e471fd-a6fc-490a-9334-8677a0e3c2ca None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:43.316389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:43.356651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bdd9a5-1653-4b2a-9ac9-6fb86e722e19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:43.357202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bdd9a5-1653-4b2a-9ac9-6fb86e722e19 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:43.357478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bdd9a5-1653-4b2a-9ac9-6fb86e722e19 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:43.357748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bdd9a5-1653-4b2a-9ac9-6fb86e722e19 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:43.369130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bdd9a5-1653-4b2a-9ac9-6fb86e722e19 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-f0bdd9a5-1653-4b2a-9ac9-6fb86e722e19 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:43.373857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:43.374863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:43.375723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 476/957] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 2 00:42:43 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:43.380954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876b8bc-ab97-435a-8e5d-8030f727d46d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:43.476413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:43.488903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876b8bc-ab97-435a-8e5d-8030f727d46d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:43.522887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22953945-9ac7-41f7-8dab-e6a05b3a1db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:43.523484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22953945-9ac7-41f7-8dab-e6a05b3a1db4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:43.523758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22953945-9ac7-41f7-8dab-e6a05b3a1db4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:43.524024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22953945-9ac7-41f7-8dab-e6a05b3a1db4 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:43.524679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22953945-9ac7-41f7-8dab-e6a05b3a1db4 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:43.524979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22953945-9ac7-41f7-8dab-e6a05b3a1db4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:43.526418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22953945-9ac7-41f7-8dab-e6a05b3a1db4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:43.534995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 482/958] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 2 00:42:43 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:43.540909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bdd9a5-1653-4b2a-9ac9-6fb86e722e19 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:43.659916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bdd9a5-1653-4b2a-9ac9-6fb86e722e19 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:43.696347 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56542a69-16f4-4ded-8dda-4ff51462214e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:43.696976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56542a69-16f4-4ded-8dda-4ff51462214e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:43.697253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56542a69-16f4-4ded-8dda-4ff51462214e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:43.697567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56542a69-16f4-4ded-8dda-4ff51462214e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:43.708396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56542a69-16f4-4ded-8dda-4ff51462214e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:43.709920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56542a69-16f4-4ded-8dda-4ff51462214e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-56542a69-16f4-4ded-8dda-4ff51462214e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:43.725658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:43.726687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:43.727614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:43.727614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:43.727614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:43.727614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 477/959] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:43.732519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22953945-9ac7-41f7-8dab-e6a05b3a1db4 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:43.833999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22953945-9ac7-41f7-8dab-e6a05b3a1db4 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:43.870479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98df3bc0-3ae7-4d92-ab0e-052d119d67c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:43.871086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98df3bc0-3ae7-4d92-ab0e-052d119d67c8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:43.871360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98df3bc0-3ae7-4d92-ab0e-052d119d67c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:43.871627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98df3bc0-3ae7-4d92-ab0e-052d119d67c8 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:43.872268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98df3bc0-3ae7-4d92-ab0e-052d119d67c8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:43.872570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98df3bc0-3ae7-4d92-ab0e-052d119d67c8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:43.873935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98df3bc0-3ae7-4d92-ab0e-052d119d67c8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:43.886991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 483/960] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:43.892533 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56542a69-16f4-4ded-8dda-4ff51462214e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:43.912073 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:44.019058 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56542a69-16f4-4ded-8dda-4ff51462214e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:44.058454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3322b5-0c73-4efe-98d3-dcc8b02fa014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:44.059083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3322b5-0c73-4efe-98d3-dcc8b02fa014 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:44.059359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3322b5-0c73-4efe-98d3-dcc8b02fa014 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:44.059633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3322b5-0c73-4efe-98d3-dcc8b02fa014 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:44.067857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3322b5-0c73-4efe-98d3-dcc8b02fa014 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:44.069361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3322b5-0c73-4efe-98d3-dcc8b02fa014 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-3c3322b5-0c73-4efe-98d3-dcc8b02fa014 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:44.080769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:44.081729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:44.082791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 478/961] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 2 00:42:43 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:44.089656 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98df3bc0-3ae7-4d92-ab0e-052d119d67c8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:44.102337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:44.203702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98df3bc0-3ae7-4d92-ab0e-052d119d67c8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:44.239851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bc177e-0d69-457f-8da5-c70f7f18e945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:44.240461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bc177e-0d69-457f-8da5-c70f7f18e945 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:44.240735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bc177e-0d69-457f-8da5-c70f7f18e945 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:44.241004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bc177e-0d69-457f-8da5-c70f7f18e945 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:44.241723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bc177e-0d69-457f-8da5-c70f7f18e945 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:44.242026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bc177e-0d69-457f-8da5-c70f7f18e945 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:44.243478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bc177e-0d69-457f-8da5-c70f7f18e945 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:44.262437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 484/962] 10.222.0.25 () {56 vars in 1149 bytes} [Sun Oct 2 00:42:44 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:44.268219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3322b5-0c73-4efe-98d3-dcc8b02fa014 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:44.275786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb31b68-b5e1-4ebc-9bff-bda118cb0e73 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:44.276058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb31b68-b5e1-4ebc-9bff-bda118cb0e73 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:44.276320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb31b68-b5e1-4ebc-9bff-bda118cb0e73 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:44.277908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 485/963] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:44.283970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afa16d3-8a33-4ce7-b1de-634518b0a3ec None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:44.284245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afa16d3-8a33-4ce7-b1de-634518b0a3ec None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:44.284513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5afa16d3-8a33-4ce7-b1de-634518b0a3ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:44.303423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5afa16d3-8a33-4ce7-b1de-634518b0a3ec None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:44.304437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5afa16d3-8a33-4ce7-b1de-634518b0a3ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:44.359470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:44.375752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:44.386325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 486/964] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:44.393747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3322b5-0c73-4efe-98d3-dcc8b02fa014 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:44.429084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d938e5-0f6c-4056-8165-878d17e50a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:44.430047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d938e5-0f6c-4056-8165-878d17e50a20 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:44.430556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d938e5-0f6c-4056-8165-878d17e50a20 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:44.431024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d938e5-0f6c-4056-8165-878d17e50a20 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:44.502133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d938e5-0f6c-4056-8165-878d17e50a20 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:44.503576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d938e5-0f6c-4056-8165-878d17e50a20 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:44.576758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:44.608284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 479/965] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:44 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:42:44.614848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afa16d3-8a33-4ce7-b1de-634518b0a3ec None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:44.706187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d938e5-0f6c-4056-8165-878d17e50a20 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:44.721161 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5afa16d3-8a33-4ce7-b1de-634518b0a3ec None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:44.759456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d9f020-35df-4d7f-ae0e-5a69a7adcacc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:44.760463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d9f020-35df-4d7f-ae0e-5a69a7adcacc None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:44.760944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d9f020-35df-4d7f-ae0e-5a69a7adcacc None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:44.761473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d9f020-35df-4d7f-ae0e-5a69a7adcacc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:44.772952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d9f020-35df-4d7f-ae0e-5a69a7adcacc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:44.774724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1d9f020-35df-4d7f-ae0e-5a69a7adcacc None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c1d9f020-35df-4d7f-ae0e-5a69a7adcacc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:44.790729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:44.791728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:44.792632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:44.792632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:44.792632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:44.794806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 487/966] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:44.800588 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d9f020-35df-4d7f-ae0e-5a69a7adcacc None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:44.826587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d938e5-0f6c-4056-8165-878d17e50a20 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:44.856791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:44.874023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc6c42f-4fc1-4e5f-81f6-a2bef04ace69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:44.874681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc6c42f-4fc1-4e5f-81f6-a2bef04ace69 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:44.874955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc6c42f-4fc1-4e5f-81f6-a2bef04ace69 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:44.875227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc6c42f-4fc1-4e5f-81f6-a2bef04ace69 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:44.875794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc6c42f-4fc1-4e5f-81f6-a2bef04ace69 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:44.877137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc6c42f-4fc1-4e5f-81f6-a2bef04ace69 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-4cc6c42f-4fc1-4e5f-81f6-a2bef04ace69 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:44.888330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:44.889364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:44.890250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:44.895398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 480/967] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:44 2022] GET /identity/v3/services/object-store => generated 93 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:44.896240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc6c42f-4fc1-4e5f-81f6-a2bef04ace69 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:44.923385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d9f020-35df-4d7f-ae0e-5a69a7adcacc None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:44.952617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:44.960035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c178193-f9f6-4f6a-9839-8bb32b99b89d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:44.960928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c178193-f9f6-4f6a-9839-8bb32b99b89d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:44.961404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c178193-f9f6-4f6a-9839-8bb32b99b89d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:44.961886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c178193-f9f6-4f6a-9839-8bb32b99b89d None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:44.962753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c178193-f9f6-4f6a-9839-8bb32b99b89d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:44.963253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c178193-f9f6-4f6a-9839-8bb32b99b89d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:44.964808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c178193-f9f6-4f6a-9839-8bb32b99b89d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:44.977929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 488/968] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:44.983234 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c178193-f9f6-4f6a-9839-8bb32b99b89d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:45.027276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc6c42f-4fc1-4e5f-81f6-a2bef04ace69 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:45.067547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01e24d3-1830-49c2-a0dc-035c4c788bc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:45.068173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01e24d3-1830-49c2-a0dc-035c4c788bc7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.068450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01e24d3-1830-49c2-a0dc-035c4c788bc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.068721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01e24d3-1830-49c2-a0dc-035c4c788bc7 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.069361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01e24d3-1830-49c2-a0dc-035c4c788bc7 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:45.069672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01e24d3-1830-49c2-a0dc-035c4c788bc7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:45.071125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01e24d3-1830-49c2-a0dc-035c4c788bc7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:45.081392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 481/969] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 2 00:42:44 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:45.082727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01e24d3-1830-49c2-a0dc-035c4c788bc7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:45.112674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c178193-f9f6-4f6a-9839-8bb32b99b89d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:45.124893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:45.152470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430f3422-3534-483b-ba43-fd3108201ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:45.153548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430f3422-3534-483b-ba43-fd3108201ecf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.154020 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430f3422-3534-483b-ba43-fd3108201ecf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.154525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430f3422-3534-483b-ba43-fd3108201ecf None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.162381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430f3422-3534-483b-ba43-fd3108201ecf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:45.164139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430f3422-3534-483b-ba43-fd3108201ecf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-430f3422-3534-483b-ba43-fd3108201ecf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:45.176871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:45.177771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:45.178724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:45.178724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:45.178724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:45.178724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:45.178724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:45.178724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:45.178724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:45.178724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:45.185630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 489/970] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:45.186723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430f3422-3534-483b-ba43-fd3108201ecf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:45.190929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01e24d3-1830-49c2-a0dc-035c4c788bc7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:45.193062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:45.231221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33eaf88-ba95-4144-a628-8b48dc992b9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:45.231823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33eaf88-ba95-4144-a628-8b48dc992b9f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.232121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33eaf88-ba95-4144-a628-8b48dc992b9f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.232370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33eaf88-ba95-4144-a628-8b48dc992b9f None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.232992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33eaf88-ba95-4144-a628-8b48dc992b9f None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:45.233322 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33eaf88-ba95-4144-a628-8b48dc992b9f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:45.234932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33eaf88-ba95-4144-a628-8b48dc992b9f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:45.244660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 482/971] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 2 00:42:45 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:45.244660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0786e88e-7e38-47f8-92ed-e5f61df81388 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.244660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0786e88e-7e38-47f8-92ed-e5f61df81388 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.244660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0786e88e-7e38-47f8-92ed-e5f61df81388 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.244660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 483/972] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:45.247174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0786e88e-7e38-47f8-92ed-e5f61df81388 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:45.296344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430f3422-3534-483b-ba43-fd3108201ecf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:45.330609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8873ffa-5079-47ba-a6ac-3edefba88702 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:45.331207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8873ffa-5079-47ba-a6ac-3edefba88702 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.331483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8873ffa-5079-47ba-a6ac-3edefba88702 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.331778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8873ffa-5079-47ba-a6ac-3edefba88702 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.332962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8873ffa-5079-47ba-a6ac-3edefba88702 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:45.333271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8873ffa-5079-47ba-a6ac-3edefba88702 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:45.335043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8873ffa-5079-47ba-a6ac-3edefba88702 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:45.347853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 490/973] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:45.349430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896e620f-fd54-4a37-837f-52d7c870bfdf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.349910 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896e620f-fd54-4a37-837f-52d7c870bfdf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.350418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896e620f-fd54-4a37-837f-52d7c870bfdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.362816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0786e88e-7e38-47f8-92ed-e5f61df81388 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:45.370550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-896e620f-fd54-4a37-837f-52d7c870bfdf None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:45.371786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-896e620f-fd54-4a37-837f-52d7c870bfdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:45.399060 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:45.407183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ecabb3-5a7c-46bf-b496-803c6ae26b13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:45.407803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ecabb3-5a7c-46bf-b496-803c6ae26b13 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.408082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ecabb3-5a7c-46bf-b496-803c6ae26b13 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.408351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ecabb3-5a7c-46bf-b496-803c6ae26b13 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.409009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ecabb3-5a7c-46bf-b496-803c6ae26b13 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:45.410411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ecabb3-5a7c-46bf-b496-803c6ae26b13 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:45.431399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 484/974] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:42:45 2022] POST /identity/v3/endpoints => generated 349 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:45.432182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ecabb3-5a7c-46bf-b496-803c6ae26b13 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:45.460014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:45.464312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 491/975] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:45.489630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10c9640-fc6f-4c3a-b214-c503371a8828 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.489944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10c9640-fc6f-4c3a-b214-c503371a8828 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.490214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10c9640-fc6f-4c3a-b214-c503371a8828 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.496186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 492/976] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:45 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:45.497615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dba802b-61f7-4c2d-8238-5d0c2e98b562 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.497897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dba802b-61f7-4c2d-8238-5d0c2e98b562 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.498208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dba802b-61f7-4c2d-8238-5d0c2e98b562 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.518806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4dba802b-61f7-4c2d-8238-5d0c2e98b562 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:45.520926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4dba802b-61f7-4c2d-8238-5d0c2e98b562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:45.570504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ecabb3-5a7c-46bf-b496-803c6ae26b13 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:45.608912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa558f96-104e-4f11-8a42-088254c68808 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:45.609525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa558f96-104e-4f11-8a42-088254c68808 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa558f96-104e-4f11-8a42-088254c68808 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.610092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa558f96-104e-4f11-8a42-088254c68808 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.610843 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa558f96-104e-4f11-8a42-088254c68808 None admin] RBAC: Adding query filter params (role.id=3d250b53a014420ab6d096fcdfc1d9e1, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=af51bf378ea2491780250fc037fa2920) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:45.611150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa558f96-104e-4f11-8a42-088254c68808 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:45.612574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa558f96-104e-4f11-8a42-088254c68808 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:45.614459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 493/977] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:45 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:45.628358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 485/978] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:45 2022] GET /identity/v3/role_assignments?user.id=af51bf378ea2491780250fc037fa2920&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=3d250b53a014420ab6d096fcdfc1d9e1 => generated 608 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:45.747772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dba802b-61f7-4c2d-8238-5d0c2e98b562 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:45.846577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:45.863875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dba802b-61f7-4c2d-8238-5d0c2e98b562 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:45.899898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71de180-3037-4f29-950b-07189fea2082 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:45.900512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71de180-3037-4f29-950b-07189fea2082 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:45.900789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71de180-3037-4f29-950b-07189fea2082 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:45.901062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71de180-3037-4f29-950b-07189fea2082 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:45.901706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71de180-3037-4f29-950b-07189fea2082 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:45.903196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71de180-3037-4f29-950b-07189fea2082 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:45.905713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa558f96-104e-4f11-8a42-088254c68808 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:45.921163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.sql.core [None req-d71de180-3037-4f29-950b-07189fea2082 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 02 00:42:45.921163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 02 00:42:45.921163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [parameters: {'user_id': '2e73096123b343fe944e3a275ec36f09', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 02 00:42:45.921163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81433) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-d71de180-3037-4f29-950b-07189fea2082 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = self._query(query) Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application conn.query(q) Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result.read() Oct 02 00:42:45.930345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 02 00:42:45.931240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application next(self.gen) Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application yield resource Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application next(self.gen) Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.session.rollback() Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.force_reraise() Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise self.value Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application session.commit() Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 02 00:42:45.932778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._prepare_impl() Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.session.flush() Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._flush(objects) Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application compat.raise_( Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application flush_context.execute() Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rec.execute(self) Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 02 00:42:45.933897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 02 00:42:45.935145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result = self._query(query) Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application conn.query(q) Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application result.read() Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 02 00:42:45.936326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [parameters: {'user_id': '2e73096123b343fe944e3a275ec36f09', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:45.943921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 02 00:42:45.945173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 02 00:42:45.946855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 02 00:42:45.946855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 02 00:42:45.946855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 02 00:42:45.946855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 02 00:42:45.946855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 02 00:42:45.946855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:45.946855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 494/979] 10.222.0.25 () {60 vars in 1131 bytes} [Sun Oct 2 00:42:45 2022] POST /identity/v3/users => generated 160 bytes in 185 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:45.946855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71de180-3037-4f29-950b-07189fea2082 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:46.012696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:46.032463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa558f96-104e-4f11-8a42-088254c68808 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:46.056098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71de180-3037-4f29-950b-07189fea2082 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:46.076716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19650026-eccd-4d6a-846f-596f087899ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:46.077631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19650026-eccd-4d6a-846f-596f087899ab None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:46.078118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19650026-eccd-4d6a-846f-596f087899ab None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:46.079044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19650026-eccd-4d6a-846f-596f087899ab None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:46.080192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19650026-eccd-4d6a-846f-596f087899ab None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:46.082728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19650026-eccd-4d6a-846f-596f087899ab None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:46.098629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9f0345-28a8-4095-9709-5fc38ad5467d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:46.099632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9f0345-28a8-4095-9709-5fc38ad5467d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:46.100162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9f0345-28a8-4095-9709-5fc38ad5467d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:46.100643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9f0345-28a8-4095-9709-5fc38ad5467d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:46.106146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 486/980] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:42:45 2022] POST /identity/v3/services => generated 242 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:42:46.116415 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9f0345-28a8-4095-9709-5fc38ad5467d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:46.119008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9f0345-28a8-4095-9709-5fc38ad5467d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bb9f0345-28a8-4095-9709-5fc38ad5467d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:46.133360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:46.134432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:46.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:46.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:46.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:46.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:46.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:46.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:46.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:46.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:46.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 02 00:42:46.135446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:46.138224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 495/981] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:42:45 2022] GET /identity/v3/users/cinder => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:46.144405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19650026-eccd-4d6a-846f-596f087899ab None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:46.267558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19650026-eccd-4d6a-846f-596f087899ab None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:46.313449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0817b6b-97dc-458b-bf56-6d41cc689190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:46.314085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0817b6b-97dc-458b-bf56-6d41cc689190 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:46.314377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0817b6b-97dc-458b-bf56-6d41cc689190 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:46.314652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0817b6b-97dc-458b-bf56-6d41cc689190 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:46.315357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0817b6b-97dc-458b-bf56-6d41cc689190 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:46.315667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0817b6b-97dc-458b-bf56-6d41cc689190 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:46.317703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0817b6b-97dc-458b-bf56-6d41cc689190 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:46.350934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:46.357006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 487/982] 10.222.0.25 () {56 vars in 1101 bytes} [Sun Oct 2 00:42:46 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:47.721885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d55b7b8-f306-484b-aaff-caa5ae008968 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:47.722463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d55b7b8-f306-484b-aaff-caa5ae008968 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:47.722463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d55b7b8-f306-484b-aaff-caa5ae008968 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:47.726754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 496/983] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:47 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:47.732086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbb60f-5204-4b6e-aa28-651987b3b327 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:47.732505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbb60f-5204-4b6e-aa28-651987b3b327 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:47.732683 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cbb60f-5204-4b6e-aa28-651987b3b327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:47.751106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-71cbb60f-5204-4b6e-aa28-651987b3b327 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:47.752287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-71cbb60f-5204-4b6e-aa28-651987b3b327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:47.824353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 488/984] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:47 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:47.895614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7363bf-3703-4414-b818-2a3ef84484f6 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:47.895950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7363bf-3703-4414-b818-2a3ef84484f6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:47.896159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7363bf-3703-4414-b818-2a3ef84484f6 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:47.897796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 497/985] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:47.902691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb7ae9c-06ab-4dae-94e1-f63b6f6102ff None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:47.902983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb7ae9c-06ab-4dae-94e1-f63b6f6102ff None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:47.903261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb7ae9c-06ab-4dae-94e1-f63b6f6102ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:47.921330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9bb7ae9c-06ab-4dae-94e1-f63b6f6102ff None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:47.922543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9bb7ae9c-06ab-4dae-94e1-f63b6f6102ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:47.988970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 489/986] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:47 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:48.098363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7363bf-3703-4414-b818-2a3ef84484f6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:48.110144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:48.202247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7363bf-3703-4414-b818-2a3ef84484f6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:48.240204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce72576-5f1d-4bc1-9de9-96522aa473a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:48.240848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce72576-5f1d-4bc1-9de9-96522aa473a0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:48.241122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce72576-5f1d-4bc1-9de9-96522aa473a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:48.241407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce72576-5f1d-4bc1-9de9-96522aa473a0 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:48.241976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce72576-5f1d-4bc1-9de9-96522aa473a0 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:48.243377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce72576-5f1d-4bc1-9de9-96522aa473a0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-4ce72576-5f1d-4bc1-9de9-96522aa473a0 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:48.252507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:48.253499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:48.254922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:48.258400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 498/987] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:48 2022] GET /identity/v3/services/object-store => generated 93 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:48.261192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb7ae9c-06ab-4dae-94e1-f63b6f6102ff None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:48.261318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce72576-5f1d-4bc1-9de9-96522aa473a0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:48.322044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:48.336681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:48.365554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb7ae9c-06ab-4dae-94e1-f63b6f6102ff None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:48.366909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce72576-5f1d-4bc1-9de9-96522aa473a0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:48.398439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ddf957-12b3-4cb1-aa98-9e86ee3797a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:48.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649a96c4-687a-41ca-8f3f-1b4161b58ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:48.399035 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ddf957-12b3-4cb1-aa98-9e86ee3797a0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:48.399307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ddf957-12b3-4cb1-aa98-9e86ee3797a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:48.399455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649a96c4-687a-41ca-8f3f-1b4161b58ee9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:48.399573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ddf957-12b3-4cb1-aa98-9e86ee3797a0 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:48.399731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649a96c4-687a-41ca-8f3f-1b4161b58ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:48.399998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649a96c4-687a-41ca-8f3f-1b4161b58ee9 None admin] PATH_INFO: `/v3/services/image` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:48.400190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ddf957-12b3-4cb1-aa98-9e86ee3797a0 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:48.400488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ddf957-12b3-4cb1-aa98-9e86ee3797a0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:48.400587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649a96c4-687a-41ca-8f3f-1b4161b58ee9 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:48.401836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ddf957-12b3-4cb1-aa98-9e86ee3797a0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:48.401934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649a96c4-687a-41ca-8f3f-1b4161b58ee9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:48.406534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 490/988] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 2 00:42:48 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-649a96c4-687a-41ca-8f3f-1b4161b58ee9 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:48.411062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:48.412056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 02 00:42:48.412907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:48.413670 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7aa777-cae5-4b9a-93ab-c04ca22f832b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:48.413757 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 499/989] 10.222.0.25 () {56 vars in 1105 bytes} [Sun Oct 2 00:42:48 2022] GET /identity/v3/services/image => generated 86 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:48.413827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7aa777-cae5-4b9a-93ab-c04ca22f832b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:48.413827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7aa777-cae5-4b9a-93ab-c04ca22f832b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:48.413958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 491/990] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:48.415044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649a96c4-687a-41ca-8f3f-1b4161b58ee9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:48.417622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7aa777-cae5-4b9a-93ab-c04ca22f832b None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:48.511034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649a96c4-687a-41ca-8f3f-1b4161b58ee9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:48.514175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7aa777-cae5-4b9a-93ab-c04ca22f832b None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:48.537954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:48.544342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b10f162-8ff8-4665-a52d-2aaa4590fcff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:48.544941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b10f162-8ff8-4665-a52d-2aaa4590fcff None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:48.545212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b10f162-8ff8-4665-a52d-2aaa4590fcff None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:48.545479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b10f162-8ff8-4665-a52d-2aaa4590fcff None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:48.546119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b10f162-8ff8-4665-a52d-2aaa4590fcff None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:48.546422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b10f162-8ff8-4665-a52d-2aaa4590fcff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:48.547778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b10f162-8ff8-4665-a52d-2aaa4590fcff None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:48.549135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bfa8db-fa0e-40de-87d7-ea4cd4fd8146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:48.549739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bfa8db-fa0e-40de-87d7-ea4cd4fd8146 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:48.550022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bfa8db-fa0e-40de-87d7-ea4cd4fd8146 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:48.550178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:48.550327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bfa8db-fa0e-40de-87d7-ea4cd4fd8146 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:48.551487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bfa8db-fa0e-40de-87d7-ea4cd4fd8146 None admin] RBAC: Adding query filter params (name=image) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:48.551790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bfa8db-fa0e-40de-87d7-ea4cd4fd8146 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:48.553124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bfa8db-fa0e-40de-87d7-ea4cd4fd8146 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:48.553692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 500/991] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 2 00:42:48 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:48.554894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2179c59e-556e-4533-87b2-bae31f40873e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:48.555324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2179c59e-556e-4533-87b2-bae31f40873e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:48.555425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2179c59e-556e-4533-87b2-bae31f40873e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:48.558664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 492/992] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:48 2022] GET /identity/v3/services?name=image => generated 122 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:48.559679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bfa8db-fa0e-40de-87d7-ea4cd4fd8146 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:48.573996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2179c59e-556e-4533-87b2-bae31f40873e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:48.575545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2179c59e-556e-4533-87b2-bae31f40873e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:48.642786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 501/993] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:48 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:48.643585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2179c59e-556e-4533-87b2-bae31f40873e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:48.657929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bfa8db-fa0e-40de-87d7-ea4cd4fd8146 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:48.695131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee9d06d-5e9b-411d-86be-83de0d085347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:48.695731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee9d06d-5e9b-411d-86be-83de0d085347 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:48.696008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee9d06d-5e9b-411d-86be-83de0d085347 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:48.696279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee9d06d-5e9b-411d-86be-83de0d085347 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:48.696983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee9d06d-5e9b-411d-86be-83de0d085347 None admin] RBAC: Adding query filter params (interface=admin, service_id=7732602dec424afdadd3388536f52caa, region_id=RegionOne) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:48.697288 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee9d06d-5e9b-411d-86be-83de0d085347 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:48.698666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee9d06d-5e9b-411d-86be-83de0d085347 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:48.703704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 493/994] 10.222.0.25 () {56 vars in 1249 bytes} [Sun Oct 2 00:42:48 2022] GET /identity/v3/endpoints?service_id=7732602dec424afdadd3388536f52caa&interface=admin®ion_id=RegionOne => generated 193 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:48.709310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee9d06d-5e9b-411d-86be-83de0d085347 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:48.757722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2179c59e-556e-4533-87b2-bae31f40873e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:48.766202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:48.790546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:48.798433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068f29c1-7f01-4a79-916d-96fb3038e087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:48.799412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068f29c1-7f01-4a79-916d-96fb3038e087 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:48.799833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068f29c1-7f01-4a79-916d-96fb3038e087 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:48.800238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068f29c1-7f01-4a79-916d-96fb3038e087 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:48.801235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068f29c1-7f01-4a79-916d-96fb3038e087 None admin] RBAC: Adding query filter params (type=image) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:48.801754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068f29c1-7f01-4a79-916d-96fb3038e087 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:48.803990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068f29c1-7f01-4a79-916d-96fb3038e087 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:48.809144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 502/995] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:48 2022] GET /identity/v3/services?type=image => generated 122 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:48.828699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee9d06d-5e9b-411d-86be-83de0d085347 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:48.866999 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b35d1b2-6c86-41e1-b45e-eac5ea494f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:48.867944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b35d1b2-6c86-41e1-b45e-eac5ea494f3d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:48.868454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b35d1b2-6c86-41e1-b45e-eac5ea494f3d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:48.868910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b35d1b2-6c86-41e1-b45e-eac5ea494f3d None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:48.869761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b35d1b2-6c86-41e1-b45e-eac5ea494f3d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:48.871456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b35d1b2-6c86-41e1-b45e-eac5ea494f3d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:48.879521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 494/996] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 2 00:42:48 2022] GET /identity/v3/services => generated 1727 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:42:48.914469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068f29c1-7f01-4a79-916d-96fb3038e087 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:49.037776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068f29c1-7f01-4a79-916d-96fb3038e087 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:49.075491 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62d7ee0-80ab-4d2b-8d0e-f40401f6840a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:49.076119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62d7ee0-80ab-4d2b-8d0e-f40401f6840a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:49.076392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62d7ee0-80ab-4d2b-8d0e-f40401f6840a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:49.076662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62d7ee0-80ab-4d2b-8d0e-f40401f6840a None admin] PATH_INFO: `/v3/services/compute` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:49.077242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62d7ee0-80ab-4d2b-8d0e-f40401f6840a None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:49.078696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62d7ee0-80ab-4d2b-8d0e-f40401f6840a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-c62d7ee0-80ab-4d2b-8d0e-f40401f6840a None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:49.088023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:49.089283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:49.090215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 503/997] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:48 2022] GET /identity/v3/services/compute => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:49.099464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b35d1b2-6c86-41e1-b45e-eac5ea494f3d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:49.201158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b35d1b2-6c86-41e1-b45e-eac5ea494f3d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:49.218742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:49.238858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e5063-08e0-436f-a571-8121c0408786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:49.239467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e5063-08e0-436f-a571-8121c0408786 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:49.239747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e5063-08e0-436f-a571-8121c0408786 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:49.240022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e5063-08e0-436f-a571-8121c0408786 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:49.240759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1e5063-08e0-436f-a571-8121c0408786 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:49.241253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1e5063-08e0-436f-a571-8121c0408786 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:49.242802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1e5063-08e0-436f-a571-8121c0408786 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:49.252050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 495/998] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:49 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:49.253544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62d7ee0-80ab-4d2b-8d0e-f40401f6840a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:49.324196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:49.367143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62d7ee0-80ab-4d2b-8d0e-f40401f6840a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:49.409695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a98cc3-0227-4129-bd6d-b1db032e025c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:49.410946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a98cc3-0227-4129-bd6d-b1db032e025c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:49.411220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a98cc3-0227-4129-bd6d-b1db032e025c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:49.411486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a98cc3-0227-4129-bd6d-b1db032e025c None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:49.412105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a98cc3-0227-4129-bd6d-b1db032e025c None admin] RBAC: Adding query filter params (type=compute) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:49.412400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a98cc3-0227-4129-bd6d-b1db032e025c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:49.413749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a98cc3-0227-4129-bd6d-b1db032e025c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:49.418667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 504/999] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:49 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:49.424384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e5063-08e0-436f-a571-8121c0408786 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:49.527534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e5063-08e0-436f-a571-8121c0408786 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:49.561630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca353ac-0242-4800-9d70-357698c3abf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:49.562255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca353ac-0242-4800-9d70-357698c3abf5 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:49.562532 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca353ac-0242-4800-9d70-357698c3abf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:49.562800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca353ac-0242-4800-9d70-357698c3abf5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:49.563487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca353ac-0242-4800-9d70-357698c3abf5 None admin] RBAC: Adding query filter params (interface=public, service_id=c4ddafaf82ba4456b8251fa4aa8ef6bd, region_id=RegionOne) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:49.563787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca353ac-0242-4800-9d70-357698c3abf5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:49.565130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca353ac-0242-4800-9d70-357698c3abf5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:49.575291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 496/1000] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 2 00:42:49 2022] GET /identity/v3/endpoints?service_id=c4ddafaf82ba4456b8251fa4aa8ef6bd&interface=public®ion_id=RegionOne => generated 194 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:49.576457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a98cc3-0227-4129-bd6d-b1db032e025c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:49.677559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a98cc3-0227-4129-bd6d-b1db032e025c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:49.699600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:49.713788 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab2a928-5cf5-41e8-b2d5-070d2d2f957e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:49.714425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab2a928-5cf5-41e8-b2d5-070d2d2f957e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:49.714697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab2a928-5cf5-41e8-b2d5-070d2d2f957e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:49.714962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab2a928-5cf5-41e8-b2d5-070d2d2f957e None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:49.715562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab2a928-5cf5-41e8-b2d5-070d2d2f957e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:49.716919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab2a928-5cf5-41e8-b2d5-070d2d2f957e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:49.730087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 505/1001] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 2 00:42:49 2022] GET /identity/v3/services => generated 1727 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:42:50.975027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8579aea2-fbbf-419f-9744-cc114a5c4b03 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:50.975568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8579aea2-fbbf-419f-9744-cc114a5c4b03 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:50.975568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8579aea2-fbbf-419f-9744-cc114a5c4b03 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:50.976942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 497/1002] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:50.982791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf96ed31-1eb0-42e0-9498-eac671ba22d9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:50.983256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf96ed31-1eb0-42e0-9498-eac671ba22d9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:50.983333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf96ed31-1eb0-42e0-9498-eac671ba22d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:51.001089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bf96ed31-1eb0-42e0-9498-eac671ba22d9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:51.002286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bf96ed31-1eb0-42e0-9498-eac671ba22d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:51.070704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d04eb5-4bd2-485c-b233-42ff3c422b06 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:51.071005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d04eb5-4bd2-485c-b233-42ff3c422b06 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:51.071307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d04eb5-4bd2-485c-b233-42ff3c422b06 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:51.076239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 498/1003] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:51.076353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 506/1004] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:50 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:51.079269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d63e2a-ef37-440d-a47d-61d6082eb180 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:51.079551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d63e2a-ef37-440d-a47d-61d6082eb180 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:51.079822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d63e2a-ef37-440d-a47d-61d6082eb180 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:51.098596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-98d63e2a-ef37-440d-a47d-61d6082eb180 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:51.099754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-98d63e2a-ef37-440d-a47d-61d6082eb180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:51.139735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:51.167405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 499/1005] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:51 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:51.343304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf96ed31-1eb0-42e0-9498-eac671ba22d9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:51.439727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d63e2a-ef37-440d-a47d-61d6082eb180 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:51.444514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf96ed31-1eb0-42e0-9498-eac671ba22d9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:51.461274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:51.481313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97076f7d-7db9-4011-9b02-a8ac394a3854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:51.481911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97076f7d-7db9-4011-9b02-a8ac394a3854 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:51.482318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97076f7d-7db9-4011-9b02-a8ac394a3854 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:51.482585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97076f7d-7db9-4011-9b02-a8ac394a3854 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:51.483241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97076f7d-7db9-4011-9b02-a8ac394a3854 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:51.484589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97076f7d-7db9-4011-9b02-a8ac394a3854 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:51.510437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 507/1006] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:42:51 2022] POST /identity/v3/services => generated 242 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:51.544134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d63e2a-ef37-440d-a47d-61d6082eb180 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:51.587426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09eb0b6-02a3-4857-b4b9-f4923baaae22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:51.588070 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09eb0b6-02a3-4857-b4b9-f4923baaae22 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:51.588342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09eb0b6-02a3-4857-b4b9-f4923baaae22 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:51.588607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09eb0b6-02a3-4857-b4b9-f4923baaae22 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:51.589183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09eb0b6-02a3-4857-b4b9-f4923baaae22 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:51.590586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09eb0b6-02a3-4857-b4b9-f4923baaae22 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-c09eb0b6-02a3-4857-b4b9-f4923baaae22 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:51.600125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:51.601173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 02 00:42:51.602014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:51.606217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 500/1007] 10.222.0.25 () {56 vars in 1126 bytes} [Sun Oct 2 00:42:51 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:51.610097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97076f7d-7db9-4011-9b02-a8ac394a3854 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:51.721504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97076f7d-7db9-4011-9b02-a8ac394a3854 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:51.756295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9414bb4b-5b4f-4666-b740-3711e2363151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:51.757184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9414bb4b-5b4f-4666-b740-3711e2363151 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:51.757457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9414bb4b-5b4f-4666-b740-3711e2363151 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:51.757721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9414bb4b-5b4f-4666-b740-3711e2363151 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:51.758471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9414bb4b-5b4f-4666-b740-3711e2363151 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:51.758770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9414bb4b-5b4f-4666-b740-3711e2363151 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:51.760182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9414bb4b-5b4f-4666-b740-3711e2363151 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:51.766047 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 508/1008] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 2 00:42:51 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:51.772571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09eb0b6-02a3-4857-b4b9-f4923baaae22 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:51.807772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:51.874608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09eb0b6-02a3-4857-b4b9-f4923baaae22 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:51.883462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead3dc43-8464-49bd-b121-f5951f1b495b None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:51.883728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead3dc43-8464-49bd-b121-f5951f1b495b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:51.884088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead3dc43-8464-49bd-b121-f5951f1b495b None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:51.885674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 509/1009] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:51.890446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38de0acc-50ec-4e47-8baa-3b1b43830cd0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:51.890818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38de0acc-50ec-4e47-8baa-3b1b43830cd0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:51.891162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38de0acc-50ec-4e47-8baa-3b1b43830cd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:51.909681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-38de0acc-50ec-4e47-8baa-3b1b43830cd0 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:51.909867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef06e06-bcc2-462e-b297-4ea58058c2f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:51.910874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef06e06-bcc2-462e-b297-4ea58058c2f0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:51.911149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef06e06-bcc2-462e-b297-4ea58058c2f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:51.911417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef06e06-bcc2-462e-b297-4ea58058c2f0 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:51.911937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-38de0acc-50ec-4e47-8baa-3b1b43830cd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:51.912031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef06e06-bcc2-462e-b297-4ea58058c2f0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:51.912319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef06e06-bcc2-462e-b297-4ea58058c2f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:51.913658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef06e06-bcc2-462e-b297-4ea58058c2f0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:51.918500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 501/1010] 10.222.0.25 () {56 vars in 1122 bytes} [Sun Oct 2 00:42:51 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:51.924252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef06e06-bcc2-462e-b297-4ea58058c2f0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:51.948998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:51.985050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 510/1011] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:51 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:52.034224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:52.038936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef06e06-bcc2-462e-b297-4ea58058c2f0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:52.073086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e094f18f-828e-4b53-a668-0c7b692630b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:52.073695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094f18f-828e-4b53-a668-0c7b692630b5 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:52.073969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094f18f-828e-4b53-a668-0c7b692630b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:52.074266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094f18f-828e-4b53-a668-0c7b692630b5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:52.074921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e094f18f-828e-4b53-a668-0c7b692630b5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:52.076268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e094f18f-828e-4b53-a668-0c7b692630b5 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:52.098093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 502/1012] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:42:51 2022] POST /identity/v3/endpoints => generated 325 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:42:52.255178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38de0acc-50ec-4e47-8baa-3b1b43830cd0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:52.355933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38de0acc-50ec-4e47-8baa-3b1b43830cd0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:52.397571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f278c4-262b-4ba2-878a-926c901e8b79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:52.398223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f278c4-262b-4ba2-878a-926c901e8b79 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:52.398498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f278c4-262b-4ba2-878a-926c901e8b79 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:52.398764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f278c4-262b-4ba2-878a-926c901e8b79 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:52.399333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f278c4-262b-4ba2-878a-926c901e8b79 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:52.400681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f278c4-262b-4ba2-878a-926c901e8b79 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-89f278c4-262b-4ba2-878a-926c901e8b79 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:52.409972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:52.411146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:52.412009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 511/1013] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:52 2022] GET /identity/v3/services/compute => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:52.418996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e094f18f-828e-4b53-a668-0c7b692630b5 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:52.519526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e094f18f-828e-4b53-a668-0c7b692630b5 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:52.552057 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de81bd3e-7dfe-4782-9159-668bd510bbea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:52.552659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de81bd3e-7dfe-4782-9159-668bd510bbea None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:52.552938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de81bd3e-7dfe-4782-9159-668bd510bbea None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:52.553206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de81bd3e-7dfe-4782-9159-668bd510bbea None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:52.553824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de81bd3e-7dfe-4782-9159-668bd510bbea None admin] RBAC: Adding query filter params (name=compute) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:52.554226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de81bd3e-7dfe-4782-9159-668bd510bbea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:52.555568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de81bd3e-7dfe-4782-9159-668bd510bbea None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:52.565073 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 503/1014] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:52 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:52.566852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f278c4-262b-4ba2-878a-926c901e8b79 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:52.600331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:52.674428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f278c4-262b-4ba2-878a-926c901e8b79 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:52.708462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987fac00-1235-42aa-8ef3-c7de2fddb326 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:52.709182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987fac00-1235-42aa-8ef3-c7de2fddb326 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:52.709552 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987fac00-1235-42aa-8ef3-c7de2fddb326 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:52.709918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987fac00-1235-42aa-8ef3-c7de2fddb326 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:52.710657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987fac00-1235-42aa-8ef3-c7de2fddb326 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:52.711050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987fac00-1235-42aa-8ef3-c7de2fddb326 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:52.712504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987fac00-1235-42aa-8ef3-c7de2fddb326 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:52.722308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 512/1015] 10.222.0.25 () {56 vars in 1112 bytes} [Sun Oct 2 00:42:52 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:52.723872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de81bd3e-7dfe-4782-9159-668bd510bbea None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:52.749140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:52.826434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de81bd3e-7dfe-4782-9159-668bd510bbea None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:52.860566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c735e75-b3d5-4060-9861-6fabd3ba5590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:52.861305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c735e75-b3d5-4060-9861-6fabd3ba5590 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:52.861675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c735e75-b3d5-4060-9861-6fabd3ba5590 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:52.862040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c735e75-b3d5-4060-9861-6fabd3ba5590 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:52.862839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c735e75-b3d5-4060-9861-6fabd3ba5590 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:52.864322 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c735e75-b3d5-4060-9861-6fabd3ba5590 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:52.894102 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 504/1016] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:42:52 2022] POST /identity/v3/endpoints => generated 334 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:42:53.682321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03273a85-9b20-4cb6-82e6-0a6fa5d944ae None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:53.682841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03273a85-9b20-4cb6-82e6-0a6fa5d944ae None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:53.682841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03273a85-9b20-4cb6-82e6-0a6fa5d944ae None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:53.684235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 513/1017] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:53.689401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce29577c-0b17-4d0f-8d23-058753e77674 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:53.689794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce29577c-0b17-4d0f-8d23-058753e77674 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:53.689955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce29577c-0b17-4d0f-8d23-058753e77674 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:53.710114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ce29577c-0b17-4d0f-8d23-058753e77674 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:53.711428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ce29577c-0b17-4d0f-8d23-058753e77674 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:53.791189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 505/1018] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:53 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:54.057610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03273a85-9b20-4cb6-82e6-0a6fa5d944ae None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:54.152804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:54.162383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03273a85-9b20-4cb6-82e6-0a6fa5d944ae None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:54.200058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c540a8a-98ed-439b-808a-9a3a28a09fc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:54.200685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c540a8a-98ed-439b-808a-9a3a28a09fc5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:54.200957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c540a8a-98ed-439b-808a-9a3a28a09fc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:54.201225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c540a8a-98ed-439b-808a-9a3a28a09fc5 None admin] PATH_INFO: `/v3/services/image` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:54.201802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c540a8a-98ed-439b-808a-9a3a28a09fc5 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:54.203194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c540a8a-98ed-439b-808a-9a3a28a09fc5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-5c540a8a-98ed-439b-808a-9a3a28a09fc5 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:54.211857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:54.212920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 02 00:42:54.213789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:54.217835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 514/1019] 10.222.0.25 () {56 vars in 1105 bytes} [Sun Oct 2 00:42:54 2022] GET /identity/v3/services/image => generated 86 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:54.218863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce29577c-0b17-4d0f-8d23-058753e77674 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:54.221921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:54.279947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d1270-5f1f-4c8f-9b3d-25a5dbca1ff4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:54.280281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d1270-5f1f-4c8f-9b3d-25a5dbca1ff4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:54.280479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0d1270-5f1f-4c8f-9b3d-25a5dbca1ff4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:54.281934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 515/1020] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:54.286890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294d8397-e9a6-46ba-90c9-8d3b95a98bfe None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:54.287113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294d8397-e9a6-46ba-90c9-8d3b95a98bfe None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:54.287389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294d8397-e9a6-46ba-90c9-8d3b95a98bfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:54.308142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-294d8397-e9a6-46ba-90c9-8d3b95a98bfe None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:54.309330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-294d8397-e9a6-46ba-90c9-8d3b95a98bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:54.323351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce29577c-0b17-4d0f-8d23-058753e77674 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:54.359502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b31659-909f-438e-a6c8-c3c6703294a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:54.360334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b31659-909f-438e-a6c8-c3c6703294a1 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:54.360616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b31659-909f-438e-a6c8-c3c6703294a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:54.360889 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b31659-909f-438e-a6c8-c3c6703294a1 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:54.361514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b31659-909f-438e-a6c8-c3c6703294a1 None admin] RBAC: Adding query filter params (name=image) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:54.361817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b31659-909f-438e-a6c8-c3c6703294a1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:54.363760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b31659-909f-438e-a6c8-c3c6703294a1 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:54.373067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 506/1021] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:54 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:54.374289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b31659-909f-438e-a6c8-c3c6703294a1 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:54.386486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 516/1022] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:54 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:54.445167 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:54.479525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b31659-909f-438e-a6c8-c3c6703294a1 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:54.513696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f7621-689b-481b-9511-848fa5cb4327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:54.514318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f7621-689b-481b-9511-848fa5cb4327 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:54.514598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f7621-689b-481b-9511-848fa5cb4327 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:54.514868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14f7621-689b-481b-9511-848fa5cb4327 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:54.515490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14f7621-689b-481b-9511-848fa5cb4327 None admin] RBAC: Adding query filter params (type=image) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:54.515792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14f7621-689b-481b-9511-848fa5cb4327 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:54.517147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14f7621-689b-481b-9511-848fa5cb4327 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:54.526761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 507/1023] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:54 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:54.527753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294d8397-e9a6-46ba-90c9-8d3b95a98bfe None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:54.614295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:54.627740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294d8397-e9a6-46ba-90c9-8d3b95a98bfe None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:54.654364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f7621-689b-481b-9511-848fa5cb4327 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:54.663974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e87c217-1b10-4f1b-becd-1ae4114f7ed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:54.664581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e87c217-1b10-4f1b-becd-1ae4114f7ed6 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:54.664855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e87c217-1b10-4f1b-becd-1ae4114f7ed6 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:54.665121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e87c217-1b10-4f1b-becd-1ae4114f7ed6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:54.665817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e87c217-1b10-4f1b-becd-1ae4114f7ed6 None admin] RBAC: Adding query filter params (interface=public, service_id=7b36399077294462a4205eb91306b2c5, region_id=RegionOne) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:54.666143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e87c217-1b10-4f1b-becd-1ae4114f7ed6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:54.667497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e87c217-1b10-4f1b-becd-1ae4114f7ed6 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:54.678035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 517/1024] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 2 00:42:54 2022] GET /identity/v3/endpoints?service_id=7b36399077294462a4205eb91306b2c5&interface=public®ion_id=RegionOne => generated 194 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:54.679065 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e87c217-1b10-4f1b-becd-1ae4114f7ed6 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:54.763716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14f7621-689b-481b-9511-848fa5cb4327 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:54.784543 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e87c217-1b10-4f1b-becd-1ae4114f7ed6 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:54.795742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:54.802880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a50496-5853-430e-bc6b-d72464f17441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:54.803622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a50496-5853-430e-bc6b-d72464f17441 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:54.803897 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a50496-5853-430e-bc6b-d72464f17441 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:54.804167 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a50496-5853-430e-bc6b-d72464f17441 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:54.805256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a50496-5853-430e-bc6b-d72464f17441 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:54.806835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70a50496-5853-430e-bc6b-d72464f17441 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:54.810217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 508/1025] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:54.814008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a50496-5853-430e-bc6b-d72464f17441 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:54.829531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac493ab-ee77-4ddb-aea3-de5352b0a148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:54.830172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac493ab-ee77-4ddb-aea3-de5352b0a148 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:54.830445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac493ab-ee77-4ddb-aea3-de5352b0a148 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:54.830714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac493ab-ee77-4ddb-aea3-de5352b0a148 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:54.831312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac493ab-ee77-4ddb-aea3-de5352b0a148 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:54.832703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac493ab-ee77-4ddb-aea3-de5352b0a148 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:54.847034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 518/1026] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 2 00:42:54 2022] GET /identity/v3/services => generated 1957 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:42:54.926560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a50496-5853-430e-bc6b-d72464f17441 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:54.963000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc093e7d-4187-46ac-8922-4867c8b69cae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:54.963613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc093e7d-4187-46ac-8922-4867c8b69cae None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:54.963886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc093e7d-4187-46ac-8922-4867c8b69cae None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:54.964152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc093e7d-4187-46ac-8922-4867c8b69cae None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:54.964809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc093e7d-4187-46ac-8922-4867c8b69cae None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:54.966312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc093e7d-4187-46ac-8922-4867c8b69cae None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:54.996266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 509/1027] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:42:54 2022] POST /identity/v3/projects => generated 311 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:42:55.176514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20429810-5689-4410-ab10-04f4b77a98c5 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:55.176838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20429810-5689-4410-ab10-04f4b77a98c5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:55.177055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20429810-5689-4410-ab10-04f4b77a98c5 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:55.178688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 519/1028] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:55.184014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94989bc9-da27-4dfe-b4da-1dc9b3428932 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:55.184302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94989bc9-da27-4dfe-b4da-1dc9b3428932 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:55.184639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94989bc9-da27-4dfe-b4da-1dc9b3428932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:55.204134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-94989bc9-da27-4dfe-b4da-1dc9b3428932 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:55.205319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-94989bc9-da27-4dfe-b4da-1dc9b3428932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:55.255806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:55.279880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 510/1029] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:55 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:42:55.547426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20429810-5689-4410-ab10-04f4b77a98c5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:55.566900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:55.652292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20429810-5689-4410-ab10-04f4b77a98c5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:55.687469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07465b72-fc5a-4877-8d81-afe424f24289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:55.688091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07465b72-fc5a-4877-8d81-afe424f24289 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:55.688363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07465b72-fc5a-4877-8d81-afe424f24289 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:55.688630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07465b72-fc5a-4877-8d81-afe424f24289 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:55.698581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07465b72-fc5a-4877-8d81-afe424f24289 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-07465b72-fc5a-4877-8d81-afe424f24289 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:42:55.703294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:55.704211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:42:55.705040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:55.709358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 520/1030] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 2 00:42:55 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:55.710632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94989bc9-da27-4dfe-b4da-1dc9b3428932 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:55.813677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94989bc9-da27-4dfe-b4da-1dc9b3428932 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:55.848292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3958b56d-6e23-4e89-b988-f8c2d5c8ffba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:55.848894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3958b56d-6e23-4e89-b988-f8c2d5c8ffba None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:55.849166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3958b56d-6e23-4e89-b988-f8c2d5c8ffba None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:55.849434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3958b56d-6e23-4e89-b988-f8c2d5c8ffba None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:55.850134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3958b56d-6e23-4e89-b988-f8c2d5c8ffba None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:55.850435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3958b56d-6e23-4e89-b988-f8c2d5c8ffba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:55.851780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3958b56d-6e23-4e89-b988-f8c2d5c8ffba None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:55.864677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 511/1031] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 2 00:42:55 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:55.866450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07465b72-fc5a-4877-8d81-afe424f24289 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:55.945783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:55.970627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07465b72-fc5a-4877-8d81-afe424f24289 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:56.005408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b3054d-b6f5-464c-94fc-6b637c914b68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:56.006021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b3054d-b6f5-464c-94fc-6b637c914b68 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:56.006583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b3054d-b6f5-464c-94fc-6b637c914b68 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:56.006849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b3054d-b6f5-464c-94fc-6b637c914b68 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:56.016696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b3054d-b6f5-464c-94fc-6b637c914b68 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:56.018140 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b3054d-b6f5-464c-94fc-6b637c914b68 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-48b3054d-b6f5-464c-94fc-6b637c914b68 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:56.035941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:56.036804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:56.042490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:56.042490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:56.042490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:56.042490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 521/1032] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:56.044758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3958b56d-6e23-4e89-b988-f8c2d5c8ffba None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:56.091198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:56.147841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3958b56d-6e23-4e89-b988-f8c2d5c8ffba None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:56.181005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b22743c-ebaf-4496-988d-c35426a3ec21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:56.181603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b22743c-ebaf-4496-988d-c35426a3ec21 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:56.181875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b22743c-ebaf-4496-988d-c35426a3ec21 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:56.182171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b22743c-ebaf-4496-988d-c35426a3ec21 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:56.182799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b22743c-ebaf-4496-988d-c35426a3ec21 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:56.183095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b22743c-ebaf-4496-988d-c35426a3ec21 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:56.184438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b22743c-ebaf-4496-988d-c35426a3ec21 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:56.196478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 512/1033] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:56.201948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b3054d-b6f5-464c-94fc-6b637c914b68 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:56.302246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b3054d-b6f5-464c-94fc-6b637c914b68 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:56.327625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:56.336836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b92016a-8a8e-442f-91cb-3c1492ca6d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:56.337455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b92016a-8a8e-442f-91cb-3c1492ca6d8c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:56.337726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b92016a-8a8e-442f-91cb-3c1492ca6d8c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:56.337991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b92016a-8a8e-442f-91cb-3c1492ca6d8c None admin] PATH_INFO: `/v3/users/nova` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:56.345317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b92016a-8a8e-442f-91cb-3c1492ca6d8c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:56.346707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b92016a-8a8e-442f-91cb-3c1492ca6d8c None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2b92016a-8a8e-442f-91cb-3c1492ca6d8c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:42:56.357984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:56.358939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:56.359976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:56.359976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:56.359976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:42:56.359976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:42:56.359976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:42:56.359976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:42:56.359976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:42:56.359976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:42:56.359976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:42:56.359976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:56.362358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 522/1034] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 2 00:42:56 2022] GET /identity/v3/users/nova => generated 82 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:56.365480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b22743c-ebaf-4496-988d-c35426a3ec21 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:56.463803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b22743c-ebaf-4496-988d-c35426a3ec21 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:56.467262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:56.498610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f3cd88-cccc-4905-b6cb-5d7458a2840e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:56.499215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f3cd88-cccc-4905-b6cb-5d7458a2840e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:56.499490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f3cd88-cccc-4905-b6cb-5d7458a2840e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:56.499760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f3cd88-cccc-4905-b6cb-5d7458a2840e None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:56.500477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f3cd88-cccc-4905-b6cb-5d7458a2840e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:56.500778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f3cd88-cccc-4905-b6cb-5d7458a2840e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:56.502227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f3cd88-cccc-4905-b6cb-5d7458a2840e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:56.524900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 513/1035] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 2 00:42:56 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:56.526470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b92016a-8a8e-442f-91cb-3c1492ca6d8c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:56.627392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b92016a-8a8e-442f-91cb-3c1492ca6d8c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:56.662750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465dd73d-660f-45ea-86dc-7a9821988faf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:56.663384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465dd73d-660f-45ea-86dc-7a9821988faf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:56.663657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465dd73d-660f-45ea-86dc-7a9821988faf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:56.663941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465dd73d-660f-45ea-86dc-7a9821988faf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:56.717676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:56.731603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465dd73d-660f-45ea-86dc-7a9821988faf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:56.733052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465dd73d-660f-45ea-86dc-7a9821988faf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:56.835497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 523/1036] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:42:56 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:42:56.837324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f3cd88-cccc-4905-b6cb-5d7458a2840e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:56.960981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f3cd88-cccc-4905-b6cb-5d7458a2840e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:56.999367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e96f9b-4719-4067-af12-90b18d90abe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:56.999997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e96f9b-4719-4067-af12-90b18d90abe4 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.000275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e96f9b-4719-4067-af12-90b18d90abe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.000544 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e96f9b-4719-4067-af12-90b18d90abe4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.011456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e96f9b-4719-4067-af12-90b18d90abe4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:57.012891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e96f9b-4719-4067-af12-90b18d90abe4 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-40e96f9b-4719-4067-af12-90b18d90abe4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:42:57.028066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:57.029049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:42:57.029958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:42:57.029958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:42:57.029958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:42:57.034633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 514/1037] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:57.034758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a230ed-5d05-4eb6-af3f-1eeb85a02094 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.034758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a230ed-5d05-4eb6-af3f-1eeb85a02094 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.035227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e96f9b-4719-4067-af12-90b18d90abe4 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:57.035303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a230ed-5d05-4eb6-af3f-1eeb85a02094 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.038386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 524/1038] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:57.041218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab8739c-d8f2-45f0-a5fe-03851e4c53ed None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.041496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab8739c-d8f2-45f0-a5fe-03851e4c53ed None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.041774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab8739c-d8f2-45f0-a5fe-03851e4c53ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.052586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:57.060718 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bab8739c-d8f2-45f0-a5fe-03851e4c53ed None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:57.061906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bab8739c-d8f2-45f0-a5fe-03851e4c53ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:57.135601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 525/1039] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:57 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:57.147932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e96f9b-4719-4067-af12-90b18d90abe4 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:57.180817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c235a4-3a33-4450-87ae-c4b6a17f4bec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:57.181427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c235a4-3a33-4450-87ae-c4b6a17f4bec None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.181706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c235a4-3a33-4450-87ae-c4b6a17f4bec None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.181976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c235a4-3a33-4450-87ae-c4b6a17f4bec None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.182656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c235a4-3a33-4450-87ae-c4b6a17f4bec None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:57.182959 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c235a4-3a33-4450-87ae-c4b6a17f4bec None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:57.184308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c235a4-3a33-4450-87ae-c4b6a17f4bec None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:57.196555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 515/1040] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:42:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:57.201917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab8739c-d8f2-45f0-a5fe-03851e4c53ed None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:57.208257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb513bb0-964c-43a7-8cc8-83a5e63a03dc None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.208535 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb513bb0-964c-43a7-8cc8-83a5e63a03dc None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.208805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb513bb0-964c-43a7-8cc8-83a5e63a03dc None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.210240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 516/1041] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:42:57.214943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6af89b0-f71c-4d76-a7fd-337845851673 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.215166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6af89b0-f71c-4d76-a7fd-337845851673 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.215435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6af89b0-f71c-4d76-a7fd-337845851673 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.222002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:57.233976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f6af89b0-f71c-4d76-a7fd-337845851673 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:57.235177 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f6af89b0-f71c-4d76-a7fd-337845851673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:57.307192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab8739c-d8f2-45f0-a5fe-03851e4c53ed None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:57.309214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 517/1042] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:57 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:42:57.341549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a274cd57-4be7-4298-93bd-1cf51aaf5b7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:57.342207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a274cd57-4be7-4298-93bd-1cf51aaf5b7d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.342485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a274cd57-4be7-4298-93bd-1cf51aaf5b7d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.342756 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a274cd57-4be7-4298-93bd-1cf51aaf5b7d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.349846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a274cd57-4be7-4298-93bd-1cf51aaf5b7d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:57.351317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a274cd57-4be7-4298-93bd-1cf51aaf5b7d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a274cd57-4be7-4298-93bd-1cf51aaf5b7d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:42:57.362298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:57.363336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:57.364728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:42:57.364728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:42:57.364728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:42:57.364728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:42:57.364728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:42:57.364728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:42:57.364728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:42:57.364728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:57.366398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 526/1043] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:42:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:42:57.369136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6af89b0-f71c-4d76-a7fd-337845851673 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:57.372644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:57.419653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a274cd57-4be7-4298-93bd-1cf51aaf5b7d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:57.470908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6af89b0-f71c-4d76-a7fd-337845851673 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:57.505180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:57.509138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55de29e-295d-4787-b5f1-cf1cd32a23fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:57.510497 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55de29e-295d-4787-b5f1-cf1cd32a23fe None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.510783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55de29e-295d-4787-b5f1-cf1cd32a23fe None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.511053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55de29e-295d-4787-b5f1-cf1cd32a23fe None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.511747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55de29e-295d-4787-b5f1-cf1cd32a23fe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:57.512054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55de29e-295d-4787-b5f1-cf1cd32a23fe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:57.513755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55de29e-295d-4787-b5f1-cf1cd32a23fe None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:57.528246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 518/1044] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:42:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:57.532891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a274cd57-4be7-4298-93bd-1cf51aaf5b7d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:57.533716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55de29e-295d-4787-b5f1-cf1cd32a23fe None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:57.567267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b756e052-ffe7-4ca2-a395-7a05d465f944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:57.568002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b756e052-ffe7-4ca2-a395-7a05d465f944 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.568291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b756e052-ffe7-4ca2-a395-7a05d465f944 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.568561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b756e052-ffe7-4ca2-a395-7a05d465f944 None admin] PATH_INFO: `/v3/services/image` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.569140 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b756e052-ffe7-4ca2-a395-7a05d465f944 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:57.570531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b756e052-ffe7-4ca2-a395-7a05d465f944 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b756e052-ffe7-4ca2-a395-7a05d465f944 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:42:57.579557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:42:57.580720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:42:57.582195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 527/1045] 10.222.0.25 () {56 vars in 1105 bytes} [Sun Oct 2 00:42:57 2022] GET /identity/v3/services/image => generated 86 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:42:57.583117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b756e052-ffe7-4ca2-a395-7a05d465f944 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:57.600439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:57.639661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55de29e-295d-4787-b5f1-cf1cd32a23fe None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:57.672746 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023c7b8a-80c8-41b9-bb96-c00f5872d9c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:57.673342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023c7b8a-80c8-41b9-bb96-c00f5872d9c0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.673617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023c7b8a-80c8-41b9-bb96-c00f5872d9c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.673888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023c7b8a-80c8-41b9-bb96-c00f5872d9c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.674638 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023c7b8a-80c8-41b9-bb96-c00f5872d9c0 None admin] RBAC: Adding query filter params (role.id=3d250b53a014420ab6d096fcdfc1d9e1, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=dad93d99b865407aa120035015f4692a) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:57.674942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023c7b8a-80c8-41b9-bb96-c00f5872d9c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:57.676894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023c7b8a-80c8-41b9-bb96-c00f5872d9c0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:57.680771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b756e052-ffe7-4ca2-a395-7a05d465f944 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:57.682021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 519/1046] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:42:57 2022] GET /identity/v3/role_assignments?user.id=dad93d99b865407aa120035015f4692a&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=3d250b53a014420ab6d096fcdfc1d9e1 => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:57.682818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023c7b8a-80c8-41b9-bb96-c00f5872d9c0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:57.732486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6c409d-8d01-42b0-9341-5c0e3401a998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:57.733228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6c409d-8d01-42b0-9341-5c0e3401a998 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.733510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6c409d-8d01-42b0-9341-5c0e3401a998 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.733784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6c409d-8d01-42b0-9341-5c0e3401a998 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.735040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6c409d-8d01-42b0-9341-5c0e3401a998 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:57.736512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6c409d-8d01-42b0-9341-5c0e3401a998 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:57.742331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 528/1047] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:57.744929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6c409d-8d01-42b0-9341-5c0e3401a998 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:57.760004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:57.801708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023c7b8a-80c8-41b9-bb96-c00f5872d9c0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:57.833727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:57.843380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca1131-e1da-495c-9299-d140d519f51f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:57.844025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca1131-e1da-495c-9299-d140d519f51f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.844260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca1131-e1da-495c-9299-d140d519f51f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.844529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ca1131-e1da-495c-9299-d140d519f51f None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.845618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ca1131-e1da-495c-9299-d140d519f51f None admin] RBAC: Adding query filter params (name=image) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:57.845924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ca1131-e1da-495c-9299-d140d519f51f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:57.848165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ca1131-e1da-495c-9299-d140d519f51f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:57.854318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 520/1048] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:57 2022] GET /identity/v3/services?name=image => generated 122 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:42:57.859858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca1131-e1da-495c-9299-d140d519f51f None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:57.863351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6c409d-8d01-42b0-9341-5c0e3401a998 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:57.904014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e13074-821f-4b7c-9805-ed94d63e4a95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:57.904679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e13074-821f-4b7c-9805-ed94d63e4a95 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:57.905475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e13074-821f-4b7c-9805-ed94d63e4a95 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:57.905759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e13074-821f-4b7c-9805-ed94d63e4a95 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:57.906447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e13074-821f-4b7c-9805-ed94d63e4a95 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:57.908366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e13074-821f-4b7c-9805-ed94d63e4a95 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:57.945021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 529/1049] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:42:57 2022] POST /identity/v3/users => generated 285 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:57.970291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ca1131-e1da-495c-9299-d140d519f51f None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:58.008441 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790030b9-eae3-4a18-af97-a2f7ed87bba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:58.009065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790030b9-eae3-4a18-af97-a2f7ed87bba3 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:58.009342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790030b9-eae3-4a18-af97-a2f7ed87bba3 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:58.009612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790030b9-eae3-4a18-af97-a2f7ed87bba3 None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:58.010274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790030b9-eae3-4a18-af97-a2f7ed87bba3 None admin] RBAC: Adding query filter params (type=image) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:42:58.010575 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790030b9-eae3-4a18-af97-a2f7ed87bba3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:58.011977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790030b9-eae3-4a18-af97-a2f7ed87bba3 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:58.017665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 521/1050] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:42:57 2022] GET /identity/v3/services?type=image => generated 350 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:42:58.023585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e13074-821f-4b7c-9805-ed94d63e4a95 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:42:58.116040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:42:58.158003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e13074-821f-4b7c-9805-ed94d63e4a95 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:42:58.197553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa0831e-d6ea-4787-8aaa-0985d17b27a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:42:58.199339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa0831e-d6ea-4787-8aaa-0985d17b27a5 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:58.199339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa0831e-d6ea-4787-8aaa-0985d17b27a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:58.199339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa0831e-d6ea-4787-8aaa-0985d17b27a5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:58.199607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa0831e-d6ea-4787-8aaa-0985d17b27a5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:42:58.200791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa0831e-d6ea-4787-8aaa-0985d17b27a5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:42:58.232991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 530/1051] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:42:58 2022] POST /identity/v3/endpoints => generated 327 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:42:59.880313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b58862d-3a25-4243-8070-39b98f1a629f None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:59.880862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b58862d-3a25-4243-8070-39b98f1a629f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:59.880862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b58862d-3a25-4243-8070-39b98f1a629f None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:59.882524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 522/1052] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:42:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:42:59.887493 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ca1b1e-e764-4d23-99cc-ea001848cc12 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:42:59.887871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ca1b1e-e764-4d23-99cc-ea001848cc12 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:42:59.888048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ca1b1e-e764-4d23-99cc-ea001848cc12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:42:59.907129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-29ca1b1e-e764-4d23-99cc-ea001848cc12 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:42:59.908309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-29ca1b1e-e764-4d23-99cc-ea001848cc12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:42:59.979707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 531/1053] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:42:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:00.208213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268181c1-a84f-496a-a4c0-5605ebe99476 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:00.208423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268181c1-a84f-496a-a4c0-5605ebe99476 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:00.208681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268181c1-a84f-496a-a4c0-5605ebe99476 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:00.210166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 523/1054] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:00.215131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4fb2cd-fd9f-4186-a5b4-9e68c717ae0b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:00.215418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4fb2cd-fd9f-4186-a5b4-9e68c717ae0b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:00.215690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4fb2cd-fd9f-4186-a5b4-9e68c717ae0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:00.233739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-de4fb2cd-fd9f-4186-a5b4-9e68c717ae0b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:00.235052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-de4fb2cd-fd9f-4186-a5b4-9e68c717ae0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:00.277792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:00.278182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268181c1-a84f-496a-a4c0-5605ebe99476 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:00.303515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 532/1055] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:43:00.339971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:00.388044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268181c1-a84f-496a-a4c0-5605ebe99476 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:00.420283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6353a480-ea05-426e-a15a-6e3f26b18e41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:00.420907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6353a480-ea05-426e-a15a-6e3f26b18e41 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:00.421185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6353a480-ea05-426e-a15a-6e3f26b18e41 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:00.421456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6353a480-ea05-426e-a15a-6e3f26b18e41 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:00.434968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6353a480-ea05-426e-a15a-6e3f26b18e41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-6353a480-ea05-426e-a15a-6e3f26b18e41 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:00.443355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:00.444303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:43:00.445147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:43:00.447748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db659dc-ab6b-4315-8f98-3f205dad3c39 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:00.448038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db659dc-ab6b-4315-8f98-3f205dad3c39 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:00.448313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db659dc-ab6b-4315-8f98-3f205dad3c39 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:00.450235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 524/1057] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 2 00:43:00 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:43:00.450659 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 533/1057] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:00.451778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6353a480-ea05-426e-a15a-6e3f26b18e41 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:00.455363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba1249f-e108-44fe-a0bf-a9cee886892c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:00.455759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba1249f-e108-44fe-a0bf-a9cee886892c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:00.456042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba1249f-e108-44fe-a0bf-a9cee886892c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:00.474844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9ba1249f-e108-44fe-a0bf-a9cee886892c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:00.475961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9ba1249f-e108-44fe-a0bf-a9cee886892c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:00.551602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 534/1058] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:43:00.561235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6353a480-ea05-426e-a15a-6e3f26b18e41 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:00.570986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba1249f-e108-44fe-a0bf-a9cee886892c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:00.578582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:00.598366 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961dc702-c6a1-4fc9-920f-250c80e7a621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:00.598981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961dc702-c6a1-4fc9-920f-250c80e7a621 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:00.599261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961dc702-c6a1-4fc9-920f-250c80e7a621 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:00.599535 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961dc702-c6a1-4fc9-920f-250c80e7a621 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:00.600193 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961dc702-c6a1-4fc9-920f-250c80e7a621 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:00.600496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961dc702-c6a1-4fc9-920f-250c80e7a621 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:00.601844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961dc702-c6a1-4fc9-920f-250c80e7a621 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:00.615619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 525/1059] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 2 00:43:00 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:00.617276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961dc702-c6a1-4fc9-920f-250c80e7a621 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:00.683494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba1249f-e108-44fe-a0bf-a9cee886892c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:00.693615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:00.720482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776078cf-1116-456c-9b41-ca4ce4be26de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:00.721103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776078cf-1116-456c-9b41-ca4ce4be26de None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:00.721384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776078cf-1116-456c-9b41-ca4ce4be26de None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:00.721653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776078cf-1116-456c-9b41-ca4ce4be26de None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:00.723380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961dc702-c6a1-4fc9-920f-250c80e7a621 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:00.733144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776078cf-1116-456c-9b41-ca4ce4be26de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-776078cf-1116-456c-9b41-ca4ce4be26de None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:00.737876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:00.739029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:00.740011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 535/1060] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:43:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:43:00.744697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776078cf-1116-456c-9b41-ca4ce4be26de None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:00.761811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd688a34-15a9-492a-8906-730da9d8fdc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:00.762464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd688a34-15a9-492a-8906-730da9d8fdc0 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:00.762740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd688a34-15a9-492a-8906-730da9d8fdc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:00.763008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd688a34-15a9-492a-8906-730da9d8fdc0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:00.774669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd688a34-15a9-492a-8906-730da9d8fdc0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:00.776130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd688a34-15a9-492a-8906-730da9d8fdc0 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-fd688a34-15a9-492a-8906-730da9d8fdc0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:43:00.790865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:00.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:43:00.792742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:43:00.792742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:43:00.792742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:43:00.796758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 526/1061] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:43:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:43:00.797588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd688a34-15a9-492a-8906-730da9d8fdc0 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:00.828763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:00.847397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776078cf-1116-456c-9b41-ca4ce4be26de None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:00.880243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594be9d0-ac1f-4a3a-a369-9a8c8a589451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:00.880835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594be9d0-ac1f-4a3a-a369-9a8c8a589451 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:00.881375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594be9d0-ac1f-4a3a-a369-9a8c8a589451 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:00.881648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594be9d0-ac1f-4a3a-a369-9a8c8a589451 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:00.882336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594be9d0-ac1f-4a3a-a369-9a8c8a589451 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:00.882635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594be9d0-ac1f-4a3a-a369-9a8c8a589451 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:00.884015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594be9d0-ac1f-4a3a-a369-9a8c8a589451 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:00.892180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 536/1062] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:43:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:43:00.893021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594be9d0-ac1f-4a3a-a369-9a8c8a589451 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:00.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd688a34-15a9-492a-8906-730da9d8fdc0 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:00.923196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:00.935466 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc1f84c-1ada-43f7-8331-b79178e507e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:00.936069 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc1f84c-1ada-43f7-8331-b79178e507e7 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:00.936350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc1f84c-1ada-43f7-8331-b79178e507e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:00.936629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc1f84c-1ada-43f7-8331-b79178e507e7 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:00.937262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc1f84c-1ada-43f7-8331-b79178e507e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:00.937567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc1f84c-1ada-43f7-8331-b79178e507e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:00.938968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc1f84c-1ada-43f7-8331-b79178e507e7 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:00.951209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 527/1063] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:43:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:00.951999 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc1f84c-1ada-43f7-8331-b79178e507e7 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:00.992763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594be9d0-ac1f-4a3a-a369-9a8c8a589451 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:01.028347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6191e6-e79b-4bc4-ae43-0504663c234a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:01.029030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6191e6-e79b-4bc4-ae43-0504663c234a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:01.029312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6191e6-e79b-4bc4-ae43-0504663c234a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:01.029583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6191e6-e79b-4bc4-ae43-0504663c234a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:01.042158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:01.052606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc1f84c-1ada-43f7-8331-b79178e507e7 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:01.088010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ebbc94-cf7a-45a8-b6f2-0a83c97e3d3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:01.088626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ebbc94-cf7a-45a8-b6f2-0a83c97e3d3d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:01.088906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ebbc94-cf7a-45a8-b6f2-0a83c97e3d3d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:01.089183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ebbc94-cf7a-45a8-b6f2-0a83c97e3d3d None admin] PATH_INFO: `/v3/users/56519e0c1db541b9ad54c8350bada2ad` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:01.096117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6191e6-e79b-4bc4-ae43-0504663c234a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:01.097668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6191e6-e79b-4bc4-ae43-0504663c234a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:01.102369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ebbc94-cf7a-45a8-b6f2-0a83c97e3d3d None admin] RBAC: Authorizing `identity:get_user(user_id=56519e0c1db541b9ad54c8350bada2ad)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:01.103788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ebbc94-cf7a-45a8-b6f2-0a83c97e3d3d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:01.105378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 528/1064] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:00 2022] GET /identity/v3/users/56519e0c1db541b9ad54c8350bada2ad => generated 285 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:01.106165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ebbc94-cf7a-45a8-b6f2-0a83c97e3d3d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:01.128558 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:01.197445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 537/1065] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:00 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:01.198309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6191e6-e79b-4bc4-ae43-0504663c234a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:01.205184 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ebbc94-cf7a-45a8-b6f2-0a83c97e3d3d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:01.237116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac22d49-aa30-487e-800b-7eae4bce1441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:01.238279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac22d49-aa30-487e-800b-7eae4bce1441 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:01.238583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac22d49-aa30-487e-800b-7eae4bce1441 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:01.238855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac22d49-aa30-487e-800b-7eae4bce1441 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:01.245807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac22d49-aa30-487e-800b-7eae4bce1441 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:01.247288 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac22d49-aa30-487e-800b-7eae4bce1441 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-cac22d49-aa30-487e-800b-7eae4bce1441 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:43:01.259464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:01.260637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 529/1066] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 2 00:43:01 2022] GET /identity/v3/users/nova => generated 82 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:01.261791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac22d49-aa30-487e-800b-7eae4bce1441 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:01.272562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:01.301066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6191e6-e79b-4bc4-ae43-0504663c234a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:01.329549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:01.336603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe31259d-9405-4611-8cae-5263e966cfce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:01.337209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe31259d-9405-4611-8cae-5263e966cfce None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:01.337482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe31259d-9405-4611-8cae-5263e966cfce None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:01.337754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe31259d-9405-4611-8cae-5263e966cfce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:01.361047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac22d49-aa30-487e-800b-7eae4bce1441 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:01.394899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a779dee-cc63-4778-994a-0f0cb155bd86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:01.395516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a779dee-cc63-4778-994a-0f0cb155bd86 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:01.395796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a779dee-cc63-4778-994a-0f0cb155bd86 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:01.396064 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a779dee-cc63-4778-994a-0f0cb155bd86 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:01.403063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe31259d-9405-4611-8cae-5263e966cfce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:01.404507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe31259d-9405-4611-8cae-5263e966cfce None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:01.406384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a779dee-cc63-4778-994a-0f0cb155bd86 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:01.407794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a779dee-cc63-4778-994a-0f0cb155bd86 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-1a779dee-cc63-4778-994a-0f0cb155bd86 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:43:01.421655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:01.422663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:43:01.424059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:43:01.424059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:43:01.424059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:43:01.424059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 530/1067] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:43:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:01.424059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a779dee-cc63-4778-994a-0f0cb155bd86 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:01.486123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:01.505398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 538/1068] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:01.506463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe31259d-9405-4611-8cae-5263e966cfce None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:01.530367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a779dee-cc63-4778-994a-0f0cb155bd86 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:01.538030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:01.564048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dca900-eb5d-48ee-beeb-7fb80e897bde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:01.564654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dca900-eb5d-48ee-beeb-7fb80e897bde None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:01.564924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dca900-eb5d-48ee-beeb-7fb80e897bde None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:01.565190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dca900-eb5d-48ee-beeb-7fb80e897bde None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:01.566414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dca900-eb5d-48ee-beeb-7fb80e897bde None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:01.566710 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dca900-eb5d-48ee-beeb-7fb80e897bde None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:01.568116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79dca900-eb5d-48ee-beeb-7fb80e897bde None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:01.585628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 531/1069] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 2 00:43:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:01.586445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dca900-eb5d-48ee-beeb-7fb80e897bde None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:01.607602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe31259d-9405-4611-8cae-5263e966cfce None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:01.641037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ff5f92-14b6-4fd8-a7ca-bce9dc4b241d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:01.641632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ff5f92-14b6-4fd8-a7ca-bce9dc4b241d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:01.641907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ff5f92-14b6-4fd8-a7ca-bce9dc4b241d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:01.642222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ff5f92-14b6-4fd8-a7ca-bce9dc4b241d None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:01.642857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ff5f92-14b6-4fd8-a7ca-bce9dc4b241d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:01.643158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ff5f92-14b6-4fd8-a7ca-bce9dc4b241d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:01.645116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ff5f92-14b6-4fd8-a7ca-bce9dc4b241d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:01.656603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 539/1070] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:43:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:01.657394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ff5f92-14b6-4fd8-a7ca-bce9dc4b241d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:01.686045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dca900-eb5d-48ee-beeb-7fb80e897bde None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:01.707916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:01.727915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23eaa4e8-0248-4139-84ac-96a243aadec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:01.728564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eaa4e8-0248-4139-84ac-96a243aadec8 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:01.728842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eaa4e8-0248-4139-84ac-96a243aadec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:01.729117 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eaa4e8-0248-4139-84ac-96a243aadec8 None admin] PATH_INFO: `/v3/projects/f3ddfa93d4764dcab99fafb891149989` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:01.730341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23eaa4e8-0248-4139-84ac-96a243aadec8 None admin] RBAC: Authorizing `identity:get_project(project_id=f3ddfa93d4764dcab99fafb891149989)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:01.731803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23eaa4e8-0248-4139-84ac-96a243aadec8 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:01.733684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 532/1071] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:01 2022] GET /identity/v3/projects/f3ddfa93d4764dcab99fafb891149989 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:01.740289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23eaa4e8-0248-4139-84ac-96a243aadec8 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:01.787200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:01.799231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ff5f92-14b6-4fd8-a7ca-bce9dc4b241d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:01.842861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cc163a-66a3-4173-8b14-bb71d49eb0aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:01.843918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cc163a-66a3-4173-8b14-bb71d49eb0aa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:01.844216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cc163a-66a3-4173-8b14-bb71d49eb0aa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:01.844495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cc163a-66a3-4173-8b14-bb71d49eb0aa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:01.858891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23eaa4e8-0248-4139-84ac-96a243aadec8 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:01.860982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cc163a-66a3-4173-8b14-bb71d49eb0aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:01.863053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cc163a-66a3-4173-8b14-bb71d49eb0aa None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a9cc163a-66a3-4173-8b14-bb71d49eb0aa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:43:01.879932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:01.880921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:43:01.882162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:43:01.882162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:43:01.882162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:01.882162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 540/1072] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:43:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:01.887706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cc163a-66a3-4173-8b14-bb71d49eb0aa None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:01.901527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df3fd92-1d42-4074-9ab8-ba0d836a4542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:01.902179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df3fd92-1d42-4074-9ab8-ba0d836a4542 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:01.903020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df3fd92-1d42-4074-9ab8-ba0d836a4542 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:01.903020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df3fd92-1d42-4074-9ab8-ba0d836a4542 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:01.903444 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df3fd92-1d42-4074-9ab8-ba0d836a4542 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=f3ddfa93d4764dcab99fafb891149989, user.id=56519e0c1db541b9ad54c8350bada2ad) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:01.903754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df3fd92-1d42-4074-9ab8-ba0d836a4542 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:01.905169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df3fd92-1d42-4074-9ab8-ba0d836a4542 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:01.910954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 533/1073] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:43:01 2022] GET /identity/v3/role_assignments?user.id=56519e0c1db541b9ad54c8350bada2ad&scope.project.id=f3ddfa93d4764dcab99fafb891149989&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:43:02.009063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cc163a-66a3-4173-8b14-bb71d49eb0aa None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:02.045823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3053ee59-b828-49a7-952d-570893f6ecb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:02.046409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3053ee59-b828-49a7-952d-570893f6ecb1 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:02.046690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3053ee59-b828-49a7-952d-570893f6ecb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:02.046962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3053ee59-b828-49a7-952d-570893f6ecb1 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:02.047603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3053ee59-b828-49a7-952d-570893f6ecb1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:02.047906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3053ee59-b828-49a7-952d-570893f6ecb1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:02.049313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3053ee59-b828-49a7-952d-570893f6ecb1 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:02.062701 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 541/1074] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:43:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:43:02.068911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df3fd92-1d42-4074-9ab8-ba0d836a4542 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:02.118735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:02.167252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df3fd92-1d42-4074-9ab8-ba0d836a4542 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:02.199906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96045b37-0ba6-4f5f-af27-4696e11351fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:02.200517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96045b37-0ba6-4f5f-af27-4696e11351fb None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:02.200789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96045b37-0ba6-4f5f-af27-4696e11351fb None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:02.201055 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96045b37-0ba6-4f5f-af27-4696e11351fb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:02.207664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96045b37-0ba6-4f5f-af27-4696e11351fb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:02.209097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96045b37-0ba6-4f5f-af27-4696e11351fb None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-96045b37-0ba6-4f5f-af27-4696e11351fb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:43:02.219868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:02.220997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:02.222631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:43:02.222631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:43:02.222631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:43:02.222631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:43:02.222631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:43:02.222631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:43:02.222631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:43:02.222631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:43:02.222631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 534/1075] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:43:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:02.227453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3053ee59-b828-49a7-952d-570893f6ecb1 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:02.230520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:02.326563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3053ee59-b828-49a7-952d-570893f6ecb1 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:02.359406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5508c25-f3f0-48a4-881e-523d7bfa1d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:02.360000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5508c25-f3f0-48a4-881e-523d7bfa1d4f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:02.360274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5508c25-f3f0-48a4-881e-523d7bfa1d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:02.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5508c25-f3f0-48a4-881e-523d7bfa1d4f None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:02.361223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5508c25-f3f0-48a4-881e-523d7bfa1d4f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:02.361525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5508c25-f3f0-48a4-881e-523d7bfa1d4f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:02.363030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5508c25-f3f0-48a4-881e-523d7bfa1d4f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:02.374375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 542/1076] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:43:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:02.381533 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96045b37-0ba6-4f5f-af27-4696e11351fb None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:02.480326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96045b37-0ba6-4f5f-af27-4696e11351fb None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:02.490184 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:02.513712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ee4ed2-9de3-4700-bbcf-0da63269b77c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:02.514384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ee4ed2-9de3-4700-bbcf-0da63269b77c None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:02.514659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ee4ed2-9de3-4700-bbcf-0da63269b77c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:02.514926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ee4ed2-9de3-4700-bbcf-0da63269b77c None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/dad93d99b865407aa120035015f4692a/roles/3d250b53a014420ab6d096fcdfc1d9e1` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:02.516988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ee4ed2-9de3-4700-bbcf-0da63269b77c None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=dad93d99b865407aa120035015f4692a, role_id=3d250b53a014420ab6d096fcdfc1d9e1)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:02.518431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ee4ed2-9de3-4700-bbcf-0da63269b77c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:02.531270 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 535/1077] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:43:02 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/dad93d99b865407aa120035015f4692a/roles/3d250b53a014420ab6d096fcdfc1d9e1 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:43:04.038638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b956dbe3-467f-4417-b1fd-32aadecac56a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:04.039084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b956dbe3-467f-4417-b1fd-32aadecac56a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:04.039194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b956dbe3-467f-4417-b1fd-32aadecac56a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:04.040501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 543/1078] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:04.045853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea442a7-3580-46bd-b8fc-d446a6cded80 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:04.046321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea442a7-3580-46bd-b8fc-d446a6cded80 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:04.046420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea442a7-3580-46bd-b8fc-d446a6cded80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:04.064691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-5ea442a7-3580-46bd-b8fc-d446a6cded80 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:04.065861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5ea442a7-3580-46bd-b8fc-d446a6cded80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:04.129180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 536/1079] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:04.400409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b956dbe3-467f-4417-b1fd-32aadecac56a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:04.464215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:04.517445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b956dbe3-467f-4417-b1fd-32aadecac56a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:04.561009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbd8dc6-d93c-41b2-91e8-c21d401e59a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:04.561653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbd8dc6-d93c-41b2-91e8-c21d401e59a4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:04.561930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbd8dc6-d93c-41b2-91e8-c21d401e59a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:04.562240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbd8dc6-d93c-41b2-91e8-c21d401e59a4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:04.574455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbd8dc6-d93c-41b2-91e8-c21d401e59a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-afbd8dc6-d93c-41b2-91e8-c21d401e59a4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:04.579283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:04.580452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:04.581969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 544/1080] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:43:04 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:04.586955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea442a7-3580-46bd-b8fc-d446a6cded80 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:04.663224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf60741-3aec-4b83-9de2-e91d3fe262f9 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:04.663430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf60741-3aec-4b83-9de2-e91d3fe262f9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:04.663662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf60741-3aec-4b83-9de2-e91d3fe262f9 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:04.666110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 545/1081] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:43:04.671226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502174df-fd1b-4d5b-a3ad-6069df96a465 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:04.671515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502174df-fd1b-4d5b-a3ad-6069df96a465 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:04.671794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-502174df-fd1b-4d5b-a3ad-6069df96a465 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:04.682807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:04.685748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea442a7-3580-46bd-b8fc-d446a6cded80 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:04.690162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-502174df-fd1b-4d5b-a3ad-6069df96a465 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:04.691869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-502174df-fd1b-4d5b-a3ad-6069df96a465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:04.721449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a0a252-6818-46f6-ab75-2085153b3117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:04.722127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a0a252-6818-46f6-ab75-2085153b3117 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:04.722407 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a0a252-6818-46f6-ab75-2085153b3117 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:04.722677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a0a252-6818-46f6-ab75-2085153b3117 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:04.723326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a0a252-6818-46f6-ab75-2085153b3117 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:04.723627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a0a252-6818-46f6-ab75-2085153b3117 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:04.724982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a0a252-6818-46f6-ab75-2085153b3117 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:04.733216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 537/1082] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:43:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:04.738886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a0a252-6818-46f6-ab75-2085153b3117 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:04.761434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 546/1083] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:43:04.836166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a0a252-6818-46f6-ab75-2085153b3117 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:04.870196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b0b266-d5f6-4a1f-a17b-fe3bb0ac6a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:04.870817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b0b266-d5f6-4a1f-a17b-fe3bb0ac6a57 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:04.871091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b0b266-d5f6-4a1f-a17b-fe3bb0ac6a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:04.871416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b0b266-d5f6-4a1f-a17b-fe3bb0ac6a57 None admin] PATH_INFO: `/v3/users/56519e0c1db541b9ad54c8350bada2ad` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:04.872815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b0b266-d5f6-4a1f-a17b-fe3bb0ac6a57 None admin] RBAC: Authorizing `identity:get_user(user_id=56519e0c1db541b9ad54c8350bada2ad)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:04.874250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b0b266-d5f6-4a1f-a17b-fe3bb0ac6a57 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:04.875802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 538/1084] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:04 2022] GET /identity/v3/users/56519e0c1db541b9ad54c8350bada2ad => generated 285 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:43:04.882447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502174df-fd1b-4d5b-a3ad-6069df96a465 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:04.932980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:04.980147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-502174df-fd1b-4d5b-a3ad-6069df96a465 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:05.013188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdf80f0-69b7-43c7-810a-cb70bd0f595b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:05.013808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdf80f0-69b7-43c7-810a-cb70bd0f595b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:05.014108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdf80f0-69b7-43c7-810a-cb70bd0f595b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:05.014384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdf80f0-69b7-43c7-810a-cb70bd0f595b None admin] PATH_INFO: `/v3/projects/f3ddfa93d4764dcab99fafb891149989` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:05.015488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdf80f0-69b7-43c7-810a-cb70bd0f595b None admin] RBAC: Authorizing `identity:get_project(project_id=f3ddfa93d4764dcab99fafb891149989)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:05.016920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdf80f0-69b7-43c7-810a-cb70bd0f595b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:05.019105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 547/1085] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:04 2022] GET /identity/v3/projects/f3ddfa93d4764dcab99fafb891149989 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:05.024890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b0b266-d5f6-4a1f-a17b-fe3bb0ac6a57 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:05.044972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdf80f0-69b7-43c7-810a-cb70bd0f595b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:05.057616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:05.129992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b0b266-d5f6-4a1f-a17b-fe3bb0ac6a57 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:05.144413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdf80f0-69b7-43c7-810a-cb70bd0f595b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:05.164403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fec3001-fdba-453c-aa40-d6cfbaed71ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:05.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fec3001-fdba-453c-aa40-d6cfbaed71ee None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:05.165346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fec3001-fdba-453c-aa40-d6cfbaed71ee None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:05.165541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:05.165666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fec3001-fdba-453c-aa40-d6cfbaed71ee None admin] PATH_INFO: `/v3/projects/f3ddfa93d4764dcab99fafb891149989/users/56519e0c1db541b9ad54c8350bada2ad/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:05.167840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fec3001-fdba-453c-aa40-d6cfbaed71ee None admin] RBAC: Authorizing `identity:create_grant(project_id=f3ddfa93d4764dcab99fafb891149989, user_id=56519e0c1db541b9ad54c8350bada2ad, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:05.169264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fec3001-fdba-453c-aa40-d6cfbaed71ee None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:05.181242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f56e86-e072-464a-abec-7afcadb3397c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:05.181976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f56e86-e072-464a-abec-7afcadb3397c None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:05.182349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f56e86-e072-464a-abec-7afcadb3397c None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:05.182560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 539/1086] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:43:05 2022] PUT /identity/v3/projects/f3ddfa93d4764dcab99fafb891149989/users/56519e0c1db541b9ad54c8350bada2ad/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:43:05.194219 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f56e86-e072-464a-abec-7afcadb3397c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:05.194430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f56e86-e072-464a-abec-7afcadb3397c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-03f56e86-e072-464a-abec-7afcadb3397c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:05.199346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:05.200520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:05.201433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 548/1087] 10.222.0.25 () {56 vars in 1120 bytes} [Sun Oct 2 00:43:05 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:43:05.208629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fec3001-fdba-453c-aa40-d6cfbaed71ee None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:05.307119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:05.318560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fec3001-fdba-453c-aa40-d6cfbaed71ee None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:05.357106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea63d5f-23dc-4f9f-b3a3-aa0e86e00849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:05.357721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea63d5f-23dc-4f9f-b3a3-aa0e86e00849 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:05.358527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea63d5f-23dc-4f9f-b3a3-aa0e86e00849 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:05.358527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea63d5f-23dc-4f9f-b3a3-aa0e86e00849 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:05.358942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea63d5f-23dc-4f9f-b3a3-aa0e86e00849 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:05.359960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea63d5f-23dc-4f9f-b3a3-aa0e86e00849 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:05.361544 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea63d5f-23dc-4f9f-b3a3-aa0e86e00849 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:05.371216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 540/1088] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 2 00:43:05 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:05.377363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f56e86-e072-464a-abec-7afcadb3397c None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:05.476094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f56e86-e072-464a-abec-7afcadb3397c None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:05.510363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b57395-36cc-409a-843f-2193114d3efa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:05.510986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b57395-36cc-409a-843f-2193114d3efa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:05.511260 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b57395-36cc-409a-843f-2193114d3efa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:05.511529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b57395-36cc-409a-843f-2193114d3efa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:05.522519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b57395-36cc-409a-843f-2193114d3efa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:05.523951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b57395-36cc-409a-843f-2193114d3efa None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-38b57395-36cc-409a-843f-2193114d3efa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:43:05.537980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:05.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:43:05.540440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:43:05.540440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:43:05.540440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:05.540440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 549/1089] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:43:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:05.544757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea63d5f-23dc-4f9f-b3a3-aa0e86e00849 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:05.642091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea63d5f-23dc-4f9f-b3a3-aa0e86e00849 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:05.675514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd00472-52a6-4098-9fa2-9506c6d6bd30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:05.676117 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd00472-52a6-4098-9fa2-9506c6d6bd30 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:05.676391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd00472-52a6-4098-9fa2-9506c6d6bd30 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:05.676662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd00472-52a6-4098-9fa2-9506c6d6bd30 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:05.677294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd00472-52a6-4098-9fa2-9506c6d6bd30 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:05.677593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd00472-52a6-4098-9fa2-9506c6d6bd30 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:05.678973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd00472-52a6-4098-9fa2-9506c6d6bd30 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:05.690023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 541/1090] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:43:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:05.696335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b57395-36cc-409a-843f-2193114d3efa None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:05.727061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:05.793005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b57395-36cc-409a-843f-2193114d3efa None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:05.825477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98abc1f4-5c9d-4598-837a-fb8fa72e5a6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:05.826120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98abc1f4-5c9d-4598-837a-fb8fa72e5a6b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:05.826399 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98abc1f4-5c9d-4598-837a-fb8fa72e5a6b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:05.826670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98abc1f4-5c9d-4598-837a-fb8fa72e5a6b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:05.835333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98abc1f4-5c9d-4598-837a-fb8fa72e5a6b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:05.836759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98abc1f4-5c9d-4598-837a-fb8fa72e5a6b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-98abc1f4-5c9d-4598-837a-fb8fa72e5a6b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:43:05.848102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:05.849221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:05.850229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 550/1091] 10.222.0.25 () {56 vars in 1093 bytes} [Sun Oct 2 00:43:05 2022] GET /identity/v3/users/nova => generated 82 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:05.854870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd00472-52a6-4098-9fa2-9506c6d6bd30 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:05.869708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:05.951423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd00472-52a6-4098-9fa2-9506c6d6bd30 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:05.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11747abb-e24d-4b15-bc58-bb279579bdec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:05.983892 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11747abb-e24d-4b15-bc58-bb279579bdec None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:05.984167 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11747abb-e24d-4b15-bc58-bb279579bdec None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:05.984435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11747abb-e24d-4b15-bc58-bb279579bdec None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:05.985142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11747abb-e24d-4b15-bc58-bb279579bdec None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:05.985438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11747abb-e24d-4b15-bc58-bb279579bdec None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:05.986865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11747abb-e24d-4b15-bc58-bb279579bdec None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:06.003674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 542/1092] 10.222.0.25 () {56 vars in 1133 bytes} [Sun Oct 2 00:43:05 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:06.009634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98abc1f4-5c9d-4598-837a-fb8fa72e5a6b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:06.105578 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:06.114901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98abc1f4-5c9d-4598-837a-fb8fa72e5a6b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:06.147166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f815b10a-ac58-43d3-993d-df3055238185 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:06.147755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f815b10a-ac58-43d3-993d-df3055238185 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:06.148030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f815b10a-ac58-43d3-993d-df3055238185 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:06.148302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f815b10a-ac58-43d3-993d-df3055238185 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:06.211778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f815b10a-ac58-43d3-993d-df3055238185 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:06.213209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f815b10a-ac58-43d3-993d-df3055238185 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:06.296669 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:06.306385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 551/1093] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:06.312131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11747abb-e24d-4b15-bc58-bb279579bdec None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:06.379377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:06.411088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11747abb-e24d-4b15-bc58-bb279579bdec None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:06.443118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cc5bba-cb09-4f2b-a7ae-8be3ef9f0904 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:06.443738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cc5bba-cb09-4f2b-a7ae-8be3ef9f0904 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:06.444014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cc5bba-cb09-4f2b-a7ae-8be3ef9f0904 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:06.444346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cc5bba-cb09-4f2b-a7ae-8be3ef9f0904 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:06.453894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cc5bba-cb09-4f2b-a7ae-8be3ef9f0904 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:06.455349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cc5bba-cb09-4f2b-a7ae-8be3ef9f0904 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-e1cc5bba-cb09-4f2b-a7ae-8be3ef9f0904 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:43:06.469857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:06.470964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:43:06.472490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:43:06.472490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:43:06.472490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:43:06.472490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 543/1094] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:43:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:06.477227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f815b10a-ac58-43d3-993d-df3055238185 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:06.571571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f815b10a-ac58-43d3-993d-df3055238185 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:06.606604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2825cc2d-1473-4ea2-b286-b95c48d5ef27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:06.607206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2825cc2d-1473-4ea2-b286-b95c48d5ef27 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:06.607479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2825cc2d-1473-4ea2-b286-b95c48d5ef27 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:06.607746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2825cc2d-1473-4ea2-b286-b95c48d5ef27 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:06.608371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2825cc2d-1473-4ea2-b286-b95c48d5ef27 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:06.608668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2825cc2d-1473-4ea2-b286-b95c48d5ef27 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:06.610007 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2825cc2d-1473-4ea2-b286-b95c48d5ef27 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:06.621117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 552/1095] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:43:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:06.626667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cc5bba-cb09-4f2b-a7ae-8be3ef9f0904 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:06.723594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cc5bba-cb09-4f2b-a7ae-8be3ef9f0904 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:06.745106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:06.757292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a64a46-216e-474e-84cc-748e76c0d068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:06.757910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a64a46-216e-474e-84cc-748e76c0d068 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:06.758223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a64a46-216e-474e-84cc-748e76c0d068 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:06.758501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a64a46-216e-474e-84cc-748e76c0d068 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:06.764934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a64a46-216e-474e-84cc-748e76c0d068 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:06.766434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a64a46-216e-474e-84cc-748e76c0d068 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-47a64a46-216e-474e-84cc-748e76c0d068 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:43:06.777884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:06.778963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:06.779890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:43:06.779890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:43:06.779890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:43:06.779890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:43:06.779890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:43:06.779890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:43:06.779890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:43:06.779890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:43:06.779890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 544/1096] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:43:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:43:06.784823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2825cc2d-1473-4ea2-b286-b95c48d5ef27 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:06.835550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:06.884525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2825cc2d-1473-4ea2-b286-b95c48d5ef27 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:06.919686 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3756623c-b59a-41e0-997a-a8f84388bafe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:06.920287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756623c-b59a-41e0-997a-a8f84388bafe None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:06.920561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756623c-b59a-41e0-997a-a8f84388bafe None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:06.920829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3756623c-b59a-41e0-997a-a8f84388bafe None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:06.921507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3756623c-b59a-41e0-997a-a8f84388bafe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:06.921811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3756623c-b59a-41e0-997a-a8f84388bafe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:06.923266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3756623c-b59a-41e0-997a-a8f84388bafe None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:06.935658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 553/1097] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:43:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:06.942180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a64a46-216e-474e-84cc-748e76c0d068 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:07.042513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a64a46-216e-474e-84cc-748e76c0d068 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:07.075444 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b6c08a4-f4c5-454f-b33f-dc29a23cf627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:07.076039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6c08a4-f4c5-454f-b33f-dc29a23cf627 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:07.076316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6c08a4-f4c5-454f-b33f-dc29a23cf627 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:07.076585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b6c08a4-f4c5-454f-b33f-dc29a23cf627 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:07.077300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6c08a4-f4c5-454f-b33f-dc29a23cf627 None admin] RBAC: Adding query filter params (role.id=3d250b53a014420ab6d096fcdfc1d9e1, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=dad93d99b865407aa120035015f4692a) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:07.077608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6c08a4-f4c5-454f-b33f-dc29a23cf627 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:07.079049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b6c08a4-f4c5-454f-b33f-dc29a23cf627 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:07.085154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 545/1098] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:43:06 2022] GET /identity/v3/role_assignments?user.id=dad93d99b865407aa120035015f4692a&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=3d250b53a014420ab6d096fcdfc1d9e1 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:07.346866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc03b60a-9ecd-458e-9569-bf3cd9b279da None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:07.347399 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc03b60a-9ecd-458e-9569-bf3cd9b279da None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:07.347399 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc03b60a-9ecd-458e-9569-bf3cd9b279da None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:07.348875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 554/1099] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:07.354105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9de4ce-784b-4ca3-9661-a5b5f549af2e None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:07.354509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9de4ce-784b-4ca3-9661-a5b5f549af2e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:07.354662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9de4ce-784b-4ca3-9661-a5b5f549af2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:07.374819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-0a9de4ce-784b-4ca3-9661-a5b5f549af2e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:07.376014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0a9de4ce-784b-4ca3-9661-a5b5f549af2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:07.419337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:07.445678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 546/1100] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:07.720227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc03b60a-9ecd-458e-9569-bf3cd9b279da None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:07.822127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc03b60a-9ecd-458e-9569-bf3cd9b279da None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:07.836586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:07.858083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbded810-e524-47ce-9510-096ec2be91fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:07.858704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbded810-e524-47ce-9510-096ec2be91fa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:07.858982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbded810-e524-47ce-9510-096ec2be91fa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:07.859253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbded810-e524-47ce-9510-096ec2be91fa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:07.870021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbded810-e524-47ce-9510-096ec2be91fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-cbded810-e524-47ce-9510-096ec2be91fa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:07.874812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:07.875863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:07.876737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 555/1101] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:43:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:07.882291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9de4ce-784b-4ca3-9661-a5b5f549af2e None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:07.983967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9de4ce-784b-4ca3-9661-a5b5f549af2e None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:08.017114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b64f6f-c283-4277-9da8-9c96dcdd55a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:08.017715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b64f6f-c283-4277-9da8-9c96dcdd55a8 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:08.017994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b64f6f-c283-4277-9da8-9c96dcdd55a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:08.018297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b64f6f-c283-4277-9da8-9c96dcdd55a8 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:08.018979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b64f6f-c283-4277-9da8-9c96dcdd55a8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:08.019318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b64f6f-c283-4277-9da8-9c96dcdd55a8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:08.020675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b64f6f-c283-4277-9da8-9c96dcdd55a8 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:08.028509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 547/1102] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:43:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:43:08.034420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbded810-e524-47ce-9510-096ec2be91fa None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:08.130452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbded810-e524-47ce-9510-096ec2be91fa None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:08.163700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0279913f-27a8-46c0-98c4-d5112f81a015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:08.164302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0279913f-27a8-46c0-98c4-d5112f81a015 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:08.164577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0279913f-27a8-46c0-98c4-d5112f81a015 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:08.164848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0279913f-27a8-46c0-98c4-d5112f81a015 None admin] PATH_INFO: `/v3/users/56519e0c1db541b9ad54c8350bada2ad` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:08.165946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0279913f-27a8-46c0-98c4-d5112f81a015 None admin] RBAC: Authorizing `identity:get_user(user_id=56519e0c1db541b9ad54c8350bada2ad)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:08.167396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0279913f-27a8-46c0-98c4-d5112f81a015 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:08.168913 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 556/1103] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:08 2022] GET /identity/v3/users/56519e0c1db541b9ad54c8350bada2ad => generated 285 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:08.174572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b64f6f-c283-4277-9da8-9c96dcdd55a8 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:08.214001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:08.273041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b64f6f-c283-4277-9da8-9c96dcdd55a8 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:08.305084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd481ea1-9745-4fa4-a175-17a1e9113a53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:08.305659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd481ea1-9745-4fa4-a175-17a1e9113a53 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:08.305932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd481ea1-9745-4fa4-a175-17a1e9113a53 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:08.306228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd481ea1-9745-4fa4-a175-17a1e9113a53 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:08.368884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd481ea1-9745-4fa4-a175-17a1e9113a53 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:08.370326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd481ea1-9745-4fa4-a175-17a1e9113a53 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:08.405462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:08.463764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 548/1104] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:08.469610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0279913f-27a8-46c0-98c4-d5112f81a015 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:08.510813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:08.566490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0279913f-27a8-46c0-98c4-d5112f81a015 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:08.598839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934872ac-8f4e-43de-bc73-730c73090cc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:08.599446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934872ac-8f4e-43de-bc73-730c73090cc9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:08.599723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934872ac-8f4e-43de-bc73-730c73090cc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:08.599994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934872ac-8f4e-43de-bc73-730c73090cc9 None admin] PATH_INFO: `/v3/projects/f3ddfa93d4764dcab99fafb891149989` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:08.601101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934872ac-8f4e-43de-bc73-730c73090cc9 None admin] RBAC: Authorizing `identity:get_project(project_id=f3ddfa93d4764dcab99fafb891149989)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:08.602564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934872ac-8f4e-43de-bc73-730c73090cc9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:08.604155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 557/1105] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:08 2022] GET /identity/v3/projects/f3ddfa93d4764dcab99fafb891149989 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:08.609966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd481ea1-9745-4fa4-a175-17a1e9113a53 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:08.705727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd481ea1-9745-4fa4-a175-17a1e9113a53 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:08.737988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2389317-1abf-4611-8ee3-b46053123326 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:08.738605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2389317-1abf-4611-8ee3-b46053123326 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:08.738883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2389317-1abf-4611-8ee3-b46053123326 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:08.739157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2389317-1abf-4611-8ee3-b46053123326 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:08.739870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2389317-1abf-4611-8ee3-b46053123326 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=f3ddfa93d4764dcab99fafb891149989, user.id=56519e0c1db541b9ad54c8350bada2ad) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:08.740179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2389317-1abf-4611-8ee3-b46053123326 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:08.741587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2389317-1abf-4611-8ee3-b46053123326 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:08.747198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 549/1106] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:43:08 2022] GET /identity/v3/role_assignments?user.id=56519e0c1db541b9ad54c8350bada2ad&scope.project.id=f3ddfa93d4764dcab99fafb891149989&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:10.892836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbafff4-7c8d-4d3c-bc29-fd7d4c1c7935 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:10.893862 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbafff4-7c8d-4d3c-bc29-fd7d4c1c7935 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:10.893862 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cbafff4-7c8d-4d3c-bc29-fd7d4c1c7935 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:10.895224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 558/1107] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:10.900325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944e400d-6397-4208-82cf-f9ae79b556c5 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:10.900691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944e400d-6397-4208-82cf-f9ae79b556c5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:10.900877 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944e400d-6397-4208-82cf-f9ae79b556c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:10.907152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:10.920297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-944e400d-6397-4208-82cf-f9ae79b556c5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:10.921476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-944e400d-6397-4208-82cf-f9ae79b556c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:10.985353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 550/1108] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:11.254252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbafff4-7c8d-4d3c-bc29-fd7d4c1c7935 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:11.351489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cbafff4-7c8d-4d3c-bc29-fd7d4c1c7935 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:11.360138 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:11.384536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd21cfe-a9b5-4131-b779-5f7a20fd52e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:11.385143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd21cfe-a9b5-4131-b779-5f7a20fd52e3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:11.385417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd21cfe-a9b5-4131-b779-5f7a20fd52e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:11.385685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd21cfe-a9b5-4131-b779-5f7a20fd52e3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:11.386797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd21cfe-a9b5-4131-b779-5f7a20fd52e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:11.388202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd21cfe-a9b5-4131-b779-5f7a20fd52e3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:11.389714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 559/1109] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:43:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:11.395599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944e400d-6397-4208-82cf-f9ae79b556c5 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:11.492188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944e400d-6397-4208-82cf-f9ae79b556c5 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:11.520811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:11.525737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d0f7a3-1335-424e-8bf1-d9e3ee232b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:11.526361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d0f7a3-1335-424e-8bf1-d9e3ee232b53 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:11.526640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d0f7a3-1335-424e-8bf1-d9e3ee232b53 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:11.526912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d0f7a3-1335-424e-8bf1-d9e3ee232b53 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:11.527558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d0f7a3-1335-424e-8bf1-d9e3ee232b53 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:11.528961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d0f7a3-1335-424e-8bf1-d9e3ee232b53 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:11.549318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 551/1110] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:43:11 2022] POST /identity/v3/users => generated 286 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:43:13.701740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb8af87-6b17-49cb-99b8-88fc784cfdf3 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:13.702309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb8af87-6b17-49cb-99b8-88fc784cfdf3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:13.702309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb8af87-6b17-49cb-99b8-88fc784cfdf3 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:13.703627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 560/1111] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:13.709499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf49f85-e285-4305-9359-20aa979e8050 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:13.709863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf49f85-e285-4305-9359-20aa979e8050 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:13.710049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf49f85-e285-4305-9359-20aa979e8050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:13.728233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8bf49f85-e285-4305-9359-20aa979e8050 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:13.729414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8bf49f85-e285-4305-9359-20aa979e8050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:13.794417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 552/1112] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:14.062221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb8af87-6b17-49cb-99b8-88fc784cfdf3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:14.159532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb8af87-6b17-49cb-99b8-88fc784cfdf3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:14.191610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6669704b-34b4-43c5-8102-442763bdc802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:14.192217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669704b-34b4-43c5-8102-442763bdc802 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:14.192492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669704b-34b4-43c5-8102-442763bdc802 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:14.192762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6669704b-34b4-43c5-8102-442763bdc802 None admin] PATH_INFO: `/v3/roles/5e19a16e5bfd4c94b1c34ae717ccfc42` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:14.194142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6669704b-34b4-43c5-8102-442763bdc802 None admin] RBAC: Authorizing `identity:get_role(role_id=5e19a16e5bfd4c94b1c34ae717ccfc42)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:14.195504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6669704b-34b4-43c5-8102-442763bdc802 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:14.196778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 561/1113] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:14 2022] GET /identity/v3/roles/5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 223 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:14.202379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf49f85-e285-4305-9359-20aa979e8050 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:14.296984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf49f85-e285-4305-9359-20aa979e8050 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:14.313059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:14.330354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a5819b-5fe7-453c-949d-838d852d6d17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:14.331702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a5819b-5fe7-453c-949d-838d852d6d17 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:14.331978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a5819b-5fe7-453c-949d-838d852d6d17 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:14.332248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a5819b-5fe7-453c-949d-838d852d6d17 None admin] PATH_INFO: `/v3/users/2d79026535a24b7bb12321ad3e08105d` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:14.344537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a5819b-5fe7-453c-949d-838d852d6d17 None admin] RBAC: Authorizing `identity:get_user(user_id=2d79026535a24b7bb12321ad3e08105d)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:14.345946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a5819b-5fe7-453c-949d-838d852d6d17 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:14.347527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 553/1114] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:14 2022] GET /identity/v3/users/2d79026535a24b7bb12321ad3e08105d => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:14.353070 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6669704b-34b4-43c5-8102-442763bdc802 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:14.397324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:14.450241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6669704b-34b4-43c5-8102-442763bdc802 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:14.482501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22503698-0f16-482b-ad09-cea1986caa61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:14.483086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22503698-0f16-482b-ad09-cea1986caa61 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:14.483362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22503698-0f16-482b-ad09-cea1986caa61 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:14.483632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22503698-0f16-482b-ad09-cea1986caa61 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:14.548245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22503698-0f16-482b-ad09-cea1986caa61 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:14.550233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22503698-0f16-482b-ad09-cea1986caa61 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:14.591318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:14.644239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 562/1115] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:14.650042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a5819b-5fe7-453c-949d-838d852d6d17 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:14.744833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a5819b-5fe7-453c-949d-838d852d6d17 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:14.776821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74af512f-d973-4c25-b2cf-beadb1dc8b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:14.777421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74af512f-d973-4c25-b2cf-beadb1dc8b9c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:14.777696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74af512f-d973-4c25-b2cf-beadb1dc8b9c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:14.777962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74af512f-d973-4c25-b2cf-beadb1dc8b9c None admin] PATH_INFO: `/v3/projects/f3ddfa93d4764dcab99fafb891149989` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:14.779074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74af512f-d973-4c25-b2cf-beadb1dc8b9c None admin] RBAC: Authorizing `identity:get_project(project_id=f3ddfa93d4764dcab99fafb891149989)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:14.780473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74af512f-d973-4c25-b2cf-beadb1dc8b9c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:14.782019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 554/1116] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:14 2022] GET /identity/v3/projects/f3ddfa93d4764dcab99fafb891149989 => generated 311 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:14.788069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22503698-0f16-482b-ad09-cea1986caa61 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:14.882968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22503698-0f16-482b-ad09-cea1986caa61 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:14.914384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6a54f-35a5-43ac-974c-4fb5a13ce836 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:14.914969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6a54f-35a5-43ac-974c-4fb5a13ce836 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:14.915245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6a54f-35a5-43ac-974c-4fb5a13ce836 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:14.915514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6a54f-35a5-43ac-974c-4fb5a13ce836 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:14.916218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf6a54f-35a5-43ac-974c-4fb5a13ce836 None admin] RBAC: Adding query filter params (role.id=5e19a16e5bfd4c94b1c34ae717ccfc42, scope.project.id=f3ddfa93d4764dcab99fafb891149989, user.id=2d79026535a24b7bb12321ad3e08105d) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:14.916521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf6a54f-35a5-43ac-974c-4fb5a13ce836 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:14.917919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf6a54f-35a5-43ac-974c-4fb5a13ce836 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:14.923028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 563/1117] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:43:14 2022] GET /identity/v3/role_assignments?user.id=2d79026535a24b7bb12321ad3e08105d&scope.project.id=f3ddfa93d4764dcab99fafb891149989&role.id=5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 259 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:17.026539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e1cd44-94ac-484e-acb1-69dcfab0c7cc None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:17.027088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e1cd44-94ac-484e-acb1-69dcfab0c7cc None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:17.027088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e1cd44-94ac-484e-acb1-69dcfab0c7cc None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:17.028415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 555/1118] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:17.034472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc9bd34-a327-4731-9a48-9f59362a836a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:17.034833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc9bd34-a327-4731-9a48-9f59362a836a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:17.035018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc9bd34-a327-4731-9a48-9f59362a836a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:17.053093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7dc9bd34-a327-4731-9a48-9f59362a836a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:17.054309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7dc9bd34-a327-4731-9a48-9f59362a836a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:17.056225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:17.119646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 564/1119] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:17.388656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e1cd44-94ac-484e-acb1-69dcfab0c7cc None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:17.434624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:17.488140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e1cd44-94ac-484e-acb1-69dcfab0c7cc None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:17.520058 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124d489c-7c15-40fc-b218-932b09143a64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:17.520670 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124d489c-7c15-40fc-b218-932b09143a64 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:17.520948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124d489c-7c15-40fc-b218-932b09143a64 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:17.521219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124d489c-7c15-40fc-b218-932b09143a64 None admin] PATH_INFO: `/v3/roles/5e19a16e5bfd4c94b1c34ae717ccfc42` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:17.522603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124d489c-7c15-40fc-b218-932b09143a64 None admin] RBAC: Authorizing `identity:get_role(role_id=5e19a16e5bfd4c94b1c34ae717ccfc42)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:17.523951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124d489c-7c15-40fc-b218-932b09143a64 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:17.525234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 556/1120] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:17 2022] GET /identity/v3/roles/5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:17.530942 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc9bd34-a327-4731-9a48-9f59362a836a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:17.625661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc9bd34-a327-4731-9a48-9f59362a836a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:17.659886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90910023-ba3c-44c9-b9cc-647c412f97e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:17.660464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90910023-ba3c-44c9-b9cc-647c412f97e0 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:17.660739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90910023-ba3c-44c9-b9cc-647c412f97e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:17.661030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90910023-ba3c-44c9-b9cc-647c412f97e0 None admin] PATH_INFO: `/v3/users/2d79026535a24b7bb12321ad3e08105d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:17.662154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90910023-ba3c-44c9-b9cc-647c412f97e0 None admin] RBAC: Authorizing `identity:get_user(user_id=2d79026535a24b7bb12321ad3e08105d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:17.663572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90910023-ba3c-44c9-b9cc-647c412f97e0 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:17.665117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 565/1121] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:17 2022] GET /identity/v3/users/2d79026535a24b7bb12321ad3e08105d => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:17.670986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124d489c-7c15-40fc-b218-932b09143a64 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:17.768440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124d489c-7c15-40fc-b218-932b09143a64 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:17.781476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:17.802605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba8ba45-4c24-4f9d-8ecd-f7e704d06d8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:17.803223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba8ba45-4c24-4f9d-8ecd-f7e704d06d8a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:17.803502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba8ba45-4c24-4f9d-8ecd-f7e704d06d8a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:17.803777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba8ba45-4c24-4f9d-8ecd-f7e704d06d8a None admin] PATH_INFO: `/v3/projects/f3ddfa93d4764dcab99fafb891149989` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:17.804903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba8ba45-4c24-4f9d-8ecd-f7e704d06d8a None admin] RBAC: Authorizing `identity:get_project(project_id=f3ddfa93d4764dcab99fafb891149989)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:17.806347 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba8ba45-4c24-4f9d-8ecd-f7e704d06d8a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:17.807949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 557/1122] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:17 2022] GET /identity/v3/projects/f3ddfa93d4764dcab99fafb891149989 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:17.814128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90910023-ba3c-44c9-b9cc-647c412f97e0 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:17.822718 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:17.910265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90910023-ba3c-44c9-b9cc-647c412f97e0 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:17.942000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2897ff-992f-4097-9fcd-7465f7f1902e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:17.942655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2897ff-992f-4097-9fcd-7465f7f1902e None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:17.942930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2897ff-992f-4097-9fcd-7465f7f1902e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:17.943198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2897ff-992f-4097-9fcd-7465f7f1902e None admin] PATH_INFO: `/v3/projects/f3ddfa93d4764dcab99fafb891149989/users/2d79026535a24b7bb12321ad3e08105d/roles/5e19a16e5bfd4c94b1c34ae717ccfc42` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:17.945291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2897ff-992f-4097-9fcd-7465f7f1902e None admin] RBAC: Authorizing `identity:create_grant(project_id=f3ddfa93d4764dcab99fafb891149989, user_id=2d79026535a24b7bb12321ad3e08105d, role_id=5e19a16e5bfd4c94b1c34ae717ccfc42)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:17.946748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2897ff-992f-4097-9fcd-7465f7f1902e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:17.958984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 566/1123] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:43:17 2022] PUT /identity/v3/projects/f3ddfa93d4764dcab99fafb891149989/users/2d79026535a24b7bb12321ad3e08105d/roles/5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:43:20.046856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ccc238-09e4-4659-b41d-028c7433b777 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:20.047300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ccc238-09e4-4659-b41d-028c7433b777 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:20.047389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ccc238-09e4-4659-b41d-028c7433b777 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:20.048691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 558/1124] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:20.054343 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b7fd0-d92c-4cde-a9fb-910508aa8d0e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:20.054769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b7fd0-d92c-4cde-a9fb-910508aa8d0e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:20.054994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b7fd0-d92c-4cde-a9fb-910508aa8d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:20.073843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cd5b7fd0-d92c-4cde-a9fb-910508aa8d0e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:20.075122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cd5b7fd0-d92c-4cde-a9fb-910508aa8d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:20.130884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:20.138606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 567/1125] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:20.402154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ccc238-09e4-4659-b41d-028c7433b777 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:20.503766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ccc238-09e4-4659-b41d-028c7433b777 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:20.536192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27b7aed-ee40-4c2c-bc79-0721c9dd1012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:20.536804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27b7aed-ee40-4c2c-bc79-0721c9dd1012 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:20.537081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27b7aed-ee40-4c2c-bc79-0721c9dd1012 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:20.537351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27b7aed-ee40-4c2c-bc79-0721c9dd1012 None admin] PATH_INFO: `/v3/roles/5e19a16e5bfd4c94b1c34ae717ccfc42` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:20.538735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27b7aed-ee40-4c2c-bc79-0721c9dd1012 None admin] RBAC: Authorizing `identity:get_role(role_id=5e19a16e5bfd4c94b1c34ae717ccfc42)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:20.540091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27b7aed-ee40-4c2c-bc79-0721c9dd1012 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:20.541371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 559/1126] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:20 2022] GET /identity/v3/roles/5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:20.547048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5b7fd0-d92c-4cde-a9fb-910508aa8d0e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:20.640189 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5b7fd0-d92c-4cde-a9fb-910508aa8d0e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:20.671469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d45b503-98bf-4b0a-a3cd-046ac74c958a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:20.672018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d45b503-98bf-4b0a-a3cd-046ac74c958a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:20.672289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d45b503-98bf-4b0a-a3cd-046ac74c958a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:20.672553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d45b503-98bf-4b0a-a3cd-046ac74c958a None admin] PATH_INFO: `/v3/users/2d79026535a24b7bb12321ad3e08105d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:20.673636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d45b503-98bf-4b0a-a3cd-046ac74c958a None admin] RBAC: Authorizing `identity:get_user(user_id=2d79026535a24b7bb12321ad3e08105d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:20.675572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d45b503-98bf-4b0a-a3cd-046ac74c958a None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:20.677077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 568/1127] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:20 2022] GET /identity/v3/users/2d79026535a24b7bb12321ad3e08105d => generated 286 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:20.682701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27b7aed-ee40-4c2c-bc79-0721c9dd1012 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:20.731358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:20.780609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27b7aed-ee40-4c2c-bc79-0721c9dd1012 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:20.812645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce587d3-cea3-49e3-a1cf-a5f607278ef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:20.813231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce587d3-cea3-49e3-a1cf-a5f607278ef4 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:20.813503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce587d3-cea3-49e3-a1cf-a5f607278ef4 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:20.813770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce587d3-cea3-49e3-a1cf-a5f607278ef4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:20.876814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce587d3-cea3-49e3-a1cf-a5f607278ef4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:20.878291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce587d3-cea3-49e3-a1cf-a5f607278ef4 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:20.922698 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:20.971214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 560/1128] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:20 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:20.977049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d45b503-98bf-4b0a-a3cd-046ac74c958a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:21.036704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:21.073922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d45b503-98bf-4b0a-a3cd-046ac74c958a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:21.105626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8db174d-ae6d-41fa-b0e2-ea83ca9cabfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:21.106261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8db174d-ae6d-41fa-b0e2-ea83ca9cabfa None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:21.106534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8db174d-ae6d-41fa-b0e2-ea83ca9cabfa None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:21.106804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8db174d-ae6d-41fa-b0e2-ea83ca9cabfa None admin] PATH_INFO: `/v3/projects/f3ddfa93d4764dcab99fafb891149989` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:21.107900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8db174d-ae6d-41fa-b0e2-ea83ca9cabfa None admin] RBAC: Authorizing `identity:get_project(project_id=f3ddfa93d4764dcab99fafb891149989)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:21.109302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8db174d-ae6d-41fa-b0e2-ea83ca9cabfa None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:21.110912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 569/1129] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:20 2022] GET /identity/v3/projects/f3ddfa93d4764dcab99fafb891149989 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:21.117157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce587d3-cea3-49e3-a1cf-a5f607278ef4 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:21.212366 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce587d3-cea3-49e3-a1cf-a5f607278ef4 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:21.244974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f2e2ce-e1ac-486e-8f96-d9e7718f2c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:21.245559 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f2e2ce-e1ac-486e-8f96-d9e7718f2c7b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:21.245834 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f2e2ce-e1ac-486e-8f96-d9e7718f2c7b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:21.246130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f2e2ce-e1ac-486e-8f96-d9e7718f2c7b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:21.246839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f2e2ce-e1ac-486e-8f96-d9e7718f2c7b None admin] RBAC: Adding query filter params (role.id=5e19a16e5bfd4c94b1c34ae717ccfc42, scope.project.id=f3ddfa93d4764dcab99fafb891149989, user.id=2d79026535a24b7bb12321ad3e08105d) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:21.247140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f2e2ce-e1ac-486e-8f96-d9e7718f2c7b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:21.248543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f2e2ce-e1ac-486e-8f96-d9e7718f2c7b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:21.254538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 561/1130] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:43:21 2022] GET /identity/v3/role_assignments?user.id=2d79026535a24b7bb12321ad3e08105d&scope.project.id=f3ddfa93d4764dcab99fafb891149989&role.id=5e19a16e5bfd4c94b1c34ae717ccfc42 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:23.369942 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f253af-eaed-42e2-901e-a766b352972c None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:23.370477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f253af-eaed-42e2-901e-a766b352972c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:23.370477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f253af-eaed-42e2-901e-a766b352972c None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:23.371747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 570/1131] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:23.377455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ea56da-d922-4a97-b3b9-2fc9c4d31827 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:23.377845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ea56da-d922-4a97-b3b9-2fc9c4d31827 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:23.377993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ea56da-d922-4a97-b3b9-2fc9c4d31827 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:23.396442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-50ea56da-d922-4a97-b3b9-2fc9c4d31827 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:23.397596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-50ea56da-d922-4a97-b3b9-2fc9c4d31827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:23.404666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:23.462785 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 562/1132] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:23.729281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f253af-eaed-42e2-901e-a766b352972c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:23.831267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f253af-eaed-42e2-901e-a766b352972c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:23.859592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:23.864622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3773841-1d3f-451c-a9fc-2254e64b2adf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:23.865235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3773841-1d3f-451c-a9fc-2254e64b2adf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:23.865512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3773841-1d3f-451c-a9fc-2254e64b2adf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:23.865780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3773841-1d3f-451c-a9fc-2254e64b2adf None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:23.866894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3773841-1d3f-451c-a9fc-2254e64b2adf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:23.868336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3773841-1d3f-451c-a9fc-2254e64b2adf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:23.869868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 571/1133] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:43:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:23.875816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ea56da-d922-4a97-b3b9-2fc9c4d31827 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:23.969101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ea56da-d922-4a97-b3b9-2fc9c4d31827 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:24.000442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aa08c80-d9f2-45be-9d98-3e1bd144a45c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:24.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa08c80-d9f2-45be-9d98-3e1bd144a45c None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:24.001299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa08c80-d9f2-45be-9d98-3e1bd144a45c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:24.001567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa08c80-d9f2-45be-9d98-3e1bd144a45c None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:24.002830 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa08c80-d9f2-45be-9d98-3e1bd144a45c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:24.004233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aa08c80-d9f2-45be-9d98-3e1bd144a45c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:24.017844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:24.030551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 563/1134] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:43:23 2022] POST /identity/v3/projects => generated 311 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:43:26.178349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40143ed-ca8a-4929-920a-cdca0ef5bc73 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:26.178861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40143ed-ca8a-4929-920a-cdca0ef5bc73 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:26.178861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40143ed-ca8a-4929-920a-cdca0ef5bc73 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:26.180203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 572/1135] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:26.185387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ac424e-d390-4195-989d-c0337e8d0e8d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:26.185751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ac424e-d390-4195-989d-c0337e8d0e8d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:26.185931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ac424e-d390-4195-989d-c0337e8d0e8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:26.204072 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f3ac424e-d390-4195-989d-c0337e8d0e8d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:26.205269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f3ac424e-d390-4195-989d-c0337e8d0e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:26.268421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 564/1136] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:26.532498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40143ed-ca8a-4929-920a-cdca0ef5bc73 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:26.629151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40143ed-ca8a-4929-920a-cdca0ef5bc73 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:26.663158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f539b3a9-2dca-46a8-b4b4-b71e72117e8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:26.663760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539b3a9-2dca-46a8-b4b4-b71e72117e8b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:26.664031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539b3a9-2dca-46a8-b4b4-b71e72117e8b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:26.664298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f539b3a9-2dca-46a8-b4b4-b71e72117e8b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:26.665381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f539b3a9-2dca-46a8-b4b4-b71e72117e8b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:26.666806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f539b3a9-2dca-46a8-b4b4-b71e72117e8b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:26.668311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 573/1137] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:43:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:26.674568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ac424e-d390-4195-989d-c0337e8d0e8d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:26.767839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ac424e-d390-4195-989d-c0337e8d0e8d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:26.789729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:26.800820 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77da017-e9f3-476c-80ea-26762d12580a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:26.801412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77da017-e9f3-476c-80ea-26762d12580a None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:26.801689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77da017-e9f3-476c-80ea-26762d12580a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:26.801956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77da017-e9f3-476c-80ea-26762d12580a None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:26.802619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77da017-e9f3-476c-80ea-26762d12580a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:26.804038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77da017-e9f3-476c-80ea-26762d12580a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:26.824672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 565/1138] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:43:26 2022] POST /identity/v3/users => generated 286 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:43:29.044082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5ac709-19d3-4b94-a305-4dcd13d316ba None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:29.044546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5ac709-19d3-4b94-a305-4dcd13d316ba None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:29.044641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5ac709-19d3-4b94-a305-4dcd13d316ba None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:29.045937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 574/1139] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:29.051163 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a689c1ec-2b14-4079-94a9-ec7f105f18fd None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:29.051578 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a689c1ec-2b14-4079-94a9-ec7f105f18fd None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:29.051720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a689c1ec-2b14-4079-94a9-ec7f105f18fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:29.069402 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a689c1ec-2b14-4079-94a9-ec7f105f18fd None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:29.070613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a689c1ec-2b14-4079-94a9-ec7f105f18fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:29.134447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 566/1140] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:29.402323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5ac709-19d3-4b94-a305-4dcd13d316ba None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:29.467234 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:29.500813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5ac709-19d3-4b94-a305-4dcd13d316ba None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:29.534177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5f1535-40ae-4629-b2e6-db73bcccb32a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:29.534784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5f1535-40ae-4629-b2e6-db73bcccb32a None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:29.535061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5f1535-40ae-4629-b2e6-db73bcccb32a None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:29.535334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5f1535-40ae-4629-b2e6-db73bcccb32a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:29.545022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5f1535-40ae-4629-b2e6-db73bcccb32a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-6e5f1535-40ae-4629-b2e6-db73bcccb32a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:29.551627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:29.552747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:29.553607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 575/1141] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:43:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:29.558597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a689c1ec-2b14-4079-94a9-ec7f105f18fd None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:29.653095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a689c1ec-2b14-4079-94a9-ec7f105f18fd None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:29.663777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:29.686856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad305970-9fff-452a-9b7c-ba46edd46bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:29.687446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad305970-9fff-452a-9b7c-ba46edd46bbb None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:29.687718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad305970-9fff-452a-9b7c-ba46edd46bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:29.687986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad305970-9fff-452a-9b7c-ba46edd46bbb None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:29.688632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad305970-9fff-452a-9b7c-ba46edd46bbb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:29.688930 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad305970-9fff-452a-9b7c-ba46edd46bbb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:29.690879 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad305970-9fff-452a-9b7c-ba46edd46bbb None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:29.698468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 567/1142] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:43:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:29.704175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5f1535-40ae-4629-b2e6-db73bcccb32a None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:29.799011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5f1535-40ae-4629-b2e6-db73bcccb32a None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:29.824624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:29.832362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656b0d6f-5086-41c9-8a52-a93cd3cb5886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:29.832970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656b0d6f-5086-41c9-8a52-a93cd3cb5886 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:29.833241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656b0d6f-5086-41c9-8a52-a93cd3cb5886 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:29.833506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656b0d6f-5086-41c9-8a52-a93cd3cb5886 None admin] PATH_INFO: `/v3/users/1b21e177e9e64173880b1d2670c10f36` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:29.846411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656b0d6f-5086-41c9-8a52-a93cd3cb5886 None admin] RBAC: Authorizing `identity:get_user(user_id=1b21e177e9e64173880b1d2670c10f36)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:29.847847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656b0d6f-5086-41c9-8a52-a93cd3cb5886 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:29.849428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 576/1143] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:29 2022] GET /identity/v3/users/1b21e177e9e64173880b1d2670c10f36 => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:43:29.855170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad305970-9fff-452a-9b7c-ba46edd46bbb None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:29.950846 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad305970-9fff-452a-9b7c-ba46edd46bbb None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:29.982422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2617916c-495a-4a20-a190-baf9d92c0eac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:29.983008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2617916c-495a-4a20-a190-baf9d92c0eac None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:29.983287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2617916c-495a-4a20-a190-baf9d92c0eac None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:29.983556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2617916c-495a-4a20-a190-baf9d92c0eac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:30.022143 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:30.048953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2617916c-495a-4a20-a190-baf9d92c0eac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:30.050681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2617916c-495a-4a20-a190-baf9d92c0eac None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:30.144071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 568/1144] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:30.150171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656b0d6f-5086-41c9-8a52-a93cd3cb5886 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:30.244638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656b0d6f-5086-41c9-8a52-a93cd3cb5886 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:30.276929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70848016-e0bb-495a-bbdd-3a681aae987e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:30.277530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70848016-e0bb-495a-bbdd-3a681aae987e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:30.277803 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70848016-e0bb-495a-bbdd-3a681aae987e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:30.278093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70848016-e0bb-495a-bbdd-3a681aae987e None admin] PATH_INFO: `/v3/projects/9ba79bd092824310835197dd9856aa9b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:30.279190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70848016-e0bb-495a-bbdd-3a681aae987e None admin] RBAC: Authorizing `identity:get_project(project_id=9ba79bd092824310835197dd9856aa9b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:30.280617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70848016-e0bb-495a-bbdd-3a681aae987e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:30.282179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 577/1145] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:30 2022] GET /identity/v3/projects/9ba79bd092824310835197dd9856aa9b => generated 311 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:30.288593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2617916c-495a-4a20-a190-baf9d92c0eac None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:30.354696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:30.384011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2617916c-495a-4a20-a190-baf9d92c0eac None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:30.416973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ddaca6-dad7-4dcc-833d-fbd33b81d41c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:30.417555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ddaca6-dad7-4dcc-833d-fbd33b81d41c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:30.417827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ddaca6-dad7-4dcc-833d-fbd33b81d41c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:30.418119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ddaca6-dad7-4dcc-833d-fbd33b81d41c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:30.418829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ddaca6-dad7-4dcc-833d-fbd33b81d41c None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=9ba79bd092824310835197dd9856aa9b, user.id=1b21e177e9e64173880b1d2670c10f36) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:30.419129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ddaca6-dad7-4dcc-833d-fbd33b81d41c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:30.420525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ddaca6-dad7-4dcc-833d-fbd33b81d41c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:30.425551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 569/1146] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:43:30 2022] GET /identity/v3/role_assignments?user.id=1b21e177e9e64173880b1d2670c10f36&scope.project.id=9ba79bd092824310835197dd9856aa9b&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:32.547433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb411d8-774d-443b-8f18-7e65da46dba9 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:32.547936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb411d8-774d-443b-8f18-7e65da46dba9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:32.548028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb411d8-774d-443b-8f18-7e65da46dba9 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:32.549361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 578/1147] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:32.554557 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66966a3-06d1-406f-a73f-de8c71e1136a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:32.554932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66966a3-06d1-406f-a73f-de8c71e1136a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:32.555146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66966a3-06d1-406f-a73f-de8c71e1136a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:32.572656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-d66966a3-06d1-406f-a73f-de8c71e1136a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:32.573825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d66966a3-06d1-406f-a73f-de8c71e1136a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:32.637849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 570/1148] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:32.906651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb411d8-774d-443b-8f18-7e65da46dba9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:32.961496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:33.006833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb411d8-774d-443b-8f18-7e65da46dba9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:33.041790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcabe86c-4ad2-44db-8766-938c39bda382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:33.042444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcabe86c-4ad2-44db-8766-938c39bda382 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:33.042740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcabe86c-4ad2-44db-8766-938c39bda382 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:33.043010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcabe86c-4ad2-44db-8766-938c39bda382 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:33.052699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcabe86c-4ad2-44db-8766-938c39bda382 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bcabe86c-4ad2-44db-8766-938c39bda382 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:33.057387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:33.058498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:33.059357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 579/1149] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:43:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:33.065104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66966a3-06d1-406f-a73f-de8c71e1136a None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:33.125840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:33.159280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66966a3-06d1-406f-a73f-de8c71e1136a None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:33.191142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259ac95-a08e-4e48-aa0f-eeeef02d04af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:33.191726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259ac95-a08e-4e48-aa0f-eeeef02d04af None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:33.192007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259ac95-a08e-4e48-aa0f-eeeef02d04af None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:33.192276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259ac95-a08e-4e48-aa0f-eeeef02d04af None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:33.192919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2259ac95-a08e-4e48-aa0f-eeeef02d04af None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:33.193217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2259ac95-a08e-4e48-aa0f-eeeef02d04af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:33.194585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2259ac95-a08e-4e48-aa0f-eeeef02d04af None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:33.202136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 571/1150] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:43:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:33.207729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcabe86c-4ad2-44db-8766-938c39bda382 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:33.302335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcabe86c-4ad2-44db-8766-938c39bda382 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:33.315406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:33.335122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd521e5-90fa-4101-84c7-263383bc4add None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:33.335722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd521e5-90fa-4101-84c7-263383bc4add None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:33.335995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd521e5-90fa-4101-84c7-263383bc4add None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:33.336263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd521e5-90fa-4101-84c7-263383bc4add None admin] PATH_INFO: `/v3/users/1b21e177e9e64173880b1d2670c10f36` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:33.337344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd521e5-90fa-4101-84c7-263383bc4add None admin] RBAC: Authorizing `identity:get_user(user_id=1b21e177e9e64173880b1d2670c10f36)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:33.338777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd521e5-90fa-4101-84c7-263383bc4add None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:33.340287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 580/1151] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:33 2022] GET /identity/v3/users/1b21e177e9e64173880b1d2670c10f36 => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:33.346092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259ac95-a08e-4e48-aa0f-eeeef02d04af None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:33.451985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259ac95-a08e-4e48-aa0f-eeeef02d04af None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:33.481480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:33.489734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc5884e-07f0-4eb1-8298-1f267d034eeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:33.490375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc5884e-07f0-4eb1-8298-1f267d034eeb None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:33.490653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc5884e-07f0-4eb1-8298-1f267d034eeb None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:33.490923 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc5884e-07f0-4eb1-8298-1f267d034eeb None admin] PATH_INFO: `/v3/projects/9ba79bd092824310835197dd9856aa9b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:33.492037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc5884e-07f0-4eb1-8298-1f267d034eeb None admin] RBAC: Authorizing `identity:get_project(project_id=9ba79bd092824310835197dd9856aa9b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:33.493451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc5884e-07f0-4eb1-8298-1f267d034eeb None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:33.495067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 572/1152] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:33 2022] GET /identity/v3/projects/9ba79bd092824310835197dd9856aa9b => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:33.500949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd521e5-90fa-4101-84c7-263383bc4add None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:33.598077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd521e5-90fa-4101-84c7-263383bc4add None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:33.630604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35a0dd3-2e93-4a7e-bc26-a0ebd7c5a3a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:33.631235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a0dd3-2e93-4a7e-bc26-a0ebd7c5a3a4 None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:33.631507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a0dd3-2e93-4a7e-bc26-a0ebd7c5a3a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:33.631773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a0dd3-2e93-4a7e-bc26-a0ebd7c5a3a4 None admin] PATH_INFO: `/v3/projects/9ba79bd092824310835197dd9856aa9b/users/1b21e177e9e64173880b1d2670c10f36/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:33.633622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35a0dd3-2e93-4a7e-bc26-a0ebd7c5a3a4 None admin] RBAC: Authorizing `identity:create_grant(project_id=9ba79bd092824310835197dd9856aa9b, user_id=1b21e177e9e64173880b1d2670c10f36, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:33.635050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35a0dd3-2e93-4a7e-bc26-a0ebd7c5a3a4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:33.647319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 581/1153] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:43:33 2022] PUT /identity/v3/projects/9ba79bd092824310835197dd9856aa9b/users/1b21e177e9e64173880b1d2670c10f36/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:43:35.737571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ae1fc0-1e6e-4f01-92b1-2ba6852d608d None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:35.738185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ae1fc0-1e6e-4f01-92b1-2ba6852d608d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:35.738185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ae1fc0-1e6e-4f01-92b1-2ba6852d608d None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:35.739470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 573/1154] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:35.744553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2563a2-0160-4337-adeb-e568ba3e27f4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:35.744989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2563a2-0160-4337-adeb-e568ba3e27f4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:35.745081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2563a2-0160-4337-adeb-e568ba3e27f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:35.763905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9f2563a2-0160-4337-adeb-e568ba3e27f4 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:35.765064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9f2563a2-0160-4337-adeb-e568ba3e27f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:35.798600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:35.831041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 582/1155] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:36.101451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ae1fc0-1e6e-4f01-92b1-2ba6852d608d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:36.199852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ae1fc0-1e6e-4f01-92b1-2ba6852d608d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:36.231611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bbdb64-ba67-448e-910a-94a0e7f3539c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:36.232214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bbdb64-ba67-448e-910a-94a0e7f3539c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:36.232493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bbdb64-ba67-448e-910a-94a0e7f3539c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:36.232764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bbdb64-ba67-448e-910a-94a0e7f3539c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:36.243449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bbdb64-ba67-448e-910a-94a0e7f3539c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-05bbdb64-ba67-448e-910a-94a0e7f3539c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:36.248159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:36.249305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:43:36.250430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 574/1156] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:43:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:36.255279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2563a2-0160-4337-adeb-e568ba3e27f4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:36.348756 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2563a2-0160-4337-adeb-e568ba3e27f4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:36.381775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e9cf32-b42e-4fa8-8c26-72d74254c4a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:36.382383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e9cf32-b42e-4fa8-8c26-72d74254c4a8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:36.382661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e9cf32-b42e-4fa8-8c26-72d74254c4a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:36.382929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e9cf32-b42e-4fa8-8c26-72d74254c4a8 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:36.383574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e9cf32-b42e-4fa8-8c26-72d74254c4a8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:36.383872 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e9cf32-b42e-4fa8-8c26-72d74254c4a8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:36.385211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e9cf32-b42e-4fa8-8c26-72d74254c4a8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:36.393359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 583/1157] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:43:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:36.399153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bbdb64-ba67-448e-910a-94a0e7f3539c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:36.452527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:36.499317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bbdb64-ba67-448e-910a-94a0e7f3539c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:36.531437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7d1f23-570f-4baf-a1a0-97c56815a6b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:36.532037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7d1f23-570f-4baf-a1a0-97c56815a6b3 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:36.532336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7d1f23-570f-4baf-a1a0-97c56815a6b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:36.532607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7d1f23-570f-4baf-a1a0-97c56815a6b3 None admin] PATH_INFO: `/v3/users/1b21e177e9e64173880b1d2670c10f36` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:36.533702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7d1f23-570f-4baf-a1a0-97c56815a6b3 None admin] RBAC: Authorizing `identity:get_user(user_id=1b21e177e9e64173880b1d2670c10f36)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:36.535139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7d1f23-570f-4baf-a1a0-97c56815a6b3 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:36.536653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 575/1158] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:36 2022] GET /identity/v3/users/1b21e177e9e64173880b1d2670c10f36 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:36.542470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e9cf32-b42e-4fa8-8c26-72d74254c4a8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:36.576130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:36.640602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e9cf32-b42e-4fa8-8c26-72d74254c4a8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:36.671422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebc48b9-d62e-4845-a8d2-e1bed68b4497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:36.672004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebc48b9-d62e-4845-a8d2-e1bed68b4497 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:36.672276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebc48b9-d62e-4845-a8d2-e1bed68b4497 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:36.672542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebc48b9-d62e-4845-a8d2-e1bed68b4497 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:36.736154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebc48b9-d62e-4845-a8d2-e1bed68b4497 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:36.737573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebc48b9-d62e-4845-a8d2-e1bed68b4497 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:36.768375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:36.831883 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 584/1159] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:36.837775 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7d1f23-570f-4baf-a1a0-97c56815a6b3 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:36.933171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7d1f23-570f-4baf-a1a0-97c56815a6b3 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:36.955334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:36.965738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9699689a-4b6b-487d-a202-37730fcd9653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:36.966376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9699689a-4b6b-487d-a202-37730fcd9653 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:36.966656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9699689a-4b6b-487d-a202-37730fcd9653 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:36.966927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9699689a-4b6b-487d-a202-37730fcd9653 None admin] PATH_INFO: `/v3/projects/9ba79bd092824310835197dd9856aa9b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:36.968012 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9699689a-4b6b-487d-a202-37730fcd9653 None admin] RBAC: Authorizing `identity:get_project(project_id=9ba79bd092824310835197dd9856aa9b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:36.969421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9699689a-4b6b-487d-a202-37730fcd9653 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:36.971562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 576/1160] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:36 2022] GET /identity/v3/projects/9ba79bd092824310835197dd9856aa9b => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:36.977534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebc48b9-d62e-4845-a8d2-e1bed68b4497 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:37.073562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebc48b9-d62e-4845-a8d2-e1bed68b4497 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:37.117326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a282f8-9ad6-4f7c-8abf-0fb3f9b6ae03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:37.117930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a282f8-9ad6-4f7c-8abf-0fb3f9b6ae03 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:37.118239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a282f8-9ad6-4f7c-8abf-0fb3f9b6ae03 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:37.118515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a282f8-9ad6-4f7c-8abf-0fb3f9b6ae03 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:37.119238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a282f8-9ad6-4f7c-8abf-0fb3f9b6ae03 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=9ba79bd092824310835197dd9856aa9b, user.id=1b21e177e9e64173880b1d2670c10f36) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:37.119544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a282f8-9ad6-4f7c-8abf-0fb3f9b6ae03 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:37.120985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a282f8-9ad6-4f7c-8abf-0fb3f9b6ae03 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:37.123010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:37.128866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 585/1161] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:43:36 2022] GET /identity/v3/role_assignments?user.id=1b21e177e9e64173880b1d2670c10f36&scope.project.id=9ba79bd092824310835197dd9856aa9b&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:39.258098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff512ea-8774-406f-bb79-c5008ba7eebe None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:39.258665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff512ea-8774-406f-bb79-c5008ba7eebe None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:39.258665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff512ea-8774-406f-bb79-c5008ba7eebe None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:39.259957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 577/1162] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:39.265116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637cbfcb-f285-4b22-ae70-a0b16b761e59 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:39.265495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637cbfcb-f285-4b22-ae70-a0b16b761e59 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:39.265652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637cbfcb-f285-4b22-ae70-a0b16b761e59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:39.283726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-637cbfcb-f285-4b22-ae70-a0b16b761e59 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:39.284824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-637cbfcb-f285-4b22-ae70-a0b16b761e59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:39.349093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 586/1163] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:39.613650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff512ea-8774-406f-bb79-c5008ba7eebe None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:39.711056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff512ea-8774-406f-bb79-c5008ba7eebe None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:39.742694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2d4bea-1cfd-4e4b-8f11-2a5befa61988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:39.743281 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2d4bea-1cfd-4e4b-8f11-2a5befa61988 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:39.743557 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2d4bea-1cfd-4e4b-8f11-2a5befa61988 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:39.743826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2d4bea-1cfd-4e4b-8f11-2a5befa61988 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:39.800045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:39.809229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2d4bea-1cfd-4e4b-8f11-2a5befa61988 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:39.810692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb2d4bea-1cfd-4e4b-8f11-2a5befa61988 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:39.904709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 578/1164] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:39.910503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637cbfcb-f285-4b22-ae70-a0b16b761e59 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:40.005726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637cbfcb-f285-4b22-ae70-a0b16b761e59 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:40.030422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:40.039071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b76a4a-cb5e-46d1-84cd-afb55e31f636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:40.039694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b76a4a-cb5e-46d1-84cd-afb55e31f636 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:40.039969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b76a4a-cb5e-46d1-84cd-afb55e31f636 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:40.040238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b76a4a-cb5e-46d1-84cd-afb55e31f636 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:40.046792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b76a4a-cb5e-46d1-84cd-afb55e31f636 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:40.048865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b76a4a-cb5e-46d1-84cd-afb55e31f636 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-33b76a4a-cb5e-46d1-84cd-afb55e31f636 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:40.059895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:40.061087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:40.062129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 587/1165] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 2 00:43:39 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:40.066897 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2d4bea-1cfd-4e4b-8f11-2a5befa61988 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:40.153163 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:40.162266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2d4bea-1cfd-4e4b-8f11-2a5befa61988 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:40.195092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d417a3b-8f43-4f29-a13e-f8e856536192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:40.195678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d417a3b-8f43-4f29-a13e-f8e856536192 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:40.195956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d417a3b-8f43-4f29-a13e-f8e856536192 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:40.196233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d417a3b-8f43-4f29-a13e-f8e856536192 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:40.196852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d417a3b-8f43-4f29-a13e-f8e856536192 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:40.197150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d417a3b-8f43-4f29-a13e-f8e856536192 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:40.198527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d417a3b-8f43-4f29-a13e-f8e856536192 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:40.205793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 579/1166] 10.222.0.25 () {56 vars in 1115 bytes} [Sun Oct 2 00:43:40 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:42.310764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db27651d-5ea1-45d1-b3db-bab408bf2324 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:42.311351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db27651d-5ea1-45d1-b3db-bab408bf2324 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:42.311351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db27651d-5ea1-45d1-b3db-bab408bf2324 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:42.312549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 588/1167] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:42.317692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b829a948-3955-4c74-86da-e3754e6a5f1b None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:42.318092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b829a948-3955-4c74-86da-e3754e6a5f1b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:42.318266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b829a948-3955-4c74-86da-e3754e6a5f1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:42.336287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-b829a948-3955-4c74-86da-e3754e6a5f1b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:42.337446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b829a948-3955-4c74-86da-e3754e6a5f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:42.401700 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 580/1168] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:42.668198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db27651d-5ea1-45d1-b3db-bab408bf2324 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:42.765393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db27651d-5ea1-45d1-b3db-bab408bf2324 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:42.798225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d916a0c4-e832-432e-aa9c-eedda27698ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:42.798812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d916a0c4-e832-432e-aa9c-eedda27698ed None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:42.799089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d916a0c4-e832-432e-aa9c-eedda27698ed None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:42.799361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d916a0c4-e832-432e-aa9c-eedda27698ed None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:42.800018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d916a0c4-e832-432e-aa9c-eedda27698ed None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:42.801375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d916a0c4-e832-432e-aa9c-eedda27698ed None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:42.818620 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 589/1169] 10.222.0.25 () {60 vars in 1138 bytes} [Sun Oct 2 00:43:42 2022] POST /identity/v3/domains => generated 267 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:43:45.027042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362ea364-21ca-4333-ba7e-6491637cc3cd None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:45.027597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362ea364-21ca-4333-ba7e-6491637cc3cd None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:45.027597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362ea364-21ca-4333-ba7e-6491637cc3cd None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:45.030543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 581/1170] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:45.035082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fccf060-2aa0-4ad9-b651-22d02b1cc798 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:45.035464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fccf060-2aa0-4ad9-b651-22d02b1cc798 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:45.035634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fccf060-2aa0-4ad9-b651-22d02b1cc798 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:45.053797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8fccf060-2aa0-4ad9-b651-22d02b1cc798 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:45.055024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8fccf060-2aa0-4ad9-b651-22d02b1cc798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:45.096865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:45.120178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 590/1171] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:43:45.388941 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362ea364-21ca-4333-ba7e-6491637cc3cd None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:45.470548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:45.485919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362ea364-21ca-4333-ba7e-6491637cc3cd None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:45.518202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2172ec91-84d9-4147-b63e-3896f57f6144 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:45.518812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2172ec91-84d9-4147-b63e-3896f57f6144 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:45.519084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2172ec91-84d9-4147-b63e-3896f57f6144 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:45.519349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2172ec91-84d9-4147-b63e-3896f57f6144 None admin] PATH_INFO: `/v3/domains/4237101012c14c3a93ca6f407d859be0` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:45.529085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2172ec91-84d9-4147-b63e-3896f57f6144 None admin] RBAC: Authorizing `identity:get_domain(domain_id=4237101012c14c3a93ca6f407d859be0)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:45.530548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2172ec91-84d9-4147-b63e-3896f57f6144 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:45.532691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 582/1172] 10.222.0.25 () {56 vars in 1183 bytes} [Sun Oct 2 00:43:45 2022] GET /identity/v3/domains/4237101012c14c3a93ca6f407d859be0 => generated 267 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:45.538620 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fccf060-2aa0-4ad9-b651-22d02b1cc798 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:45.633996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fccf060-2aa0-4ad9-b651-22d02b1cc798 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:45.668995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31440af5-5f7f-4145-bb7f-1f5c727f06e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:45.669525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31440af5-5f7f-4145-bb7f-1f5c727f06e5 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:45.669796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31440af5-5f7f-4145-bb7f-1f5c727f06e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:45.670080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31440af5-5f7f-4145-bb7f-1f5c727f06e5 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:45.670730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31440af5-5f7f-4145-bb7f-1f5c727f06e5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:45.672133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31440af5-5f7f-4145-bb7f-1f5c727f06e5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:45.697593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 591/1173] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:43:45 2022] POST /identity/v3/projects => generated 361 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:43:47.840608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951b0478-660e-4717-bc8f-e66f24e9f59a None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:47.841166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951b0478-660e-4717-bc8f-e66f24e9f59a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:47.841166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951b0478-660e-4717-bc8f-e66f24e9f59a None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:47.842486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 583/1174] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:47.847919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec345e62-a8d0-47b6-8b82-f90fd80bcc94 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:47.848348 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec345e62-a8d0-47b6-8b82-f90fd80bcc94 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:47.848471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec345e62-a8d0-47b6-8b82-f90fd80bcc94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:47.866674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ec345e62-a8d0-47b6-8b82-f90fd80bcc94 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:47.867849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ec345e62-a8d0-47b6-8b82-f90fd80bcc94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:47.890931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:47.932299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 592/1175] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:48.202641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951b0478-660e-4717-bc8f-e66f24e9f59a None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:48.302283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951b0478-660e-4717-bc8f-e66f24e9f59a None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:48.334419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31868c6b-c569-46ac-a167-0393d6ae049e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:48.335037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31868c6b-c569-46ac-a167-0393d6ae049e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:48.335315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31868c6b-c569-46ac-a167-0393d6ae049e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:48.335587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31868c6b-c569-46ac-a167-0393d6ae049e None admin] PATH_INFO: `/v3/domains/4237101012c14c3a93ca6f407d859be0` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:48.336684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31868c6b-c569-46ac-a167-0393d6ae049e None admin] RBAC: Authorizing `identity:get_domain(domain_id=4237101012c14c3a93ca6f407d859be0)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:48.338140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31868c6b-c569-46ac-a167-0393d6ae049e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:48.339700 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 584/1176] 10.222.0.25 () {56 vars in 1183 bytes} [Sun Oct 2 00:43:48 2022] GET /identity/v3/domains/4237101012c14c3a93ca6f407d859be0 => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:48.347722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec345e62-a8d0-47b6-8b82-f90fd80bcc94 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:48.441917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec345e62-a8d0-47b6-8b82-f90fd80bcc94 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:48.474222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0d1261-e4a3-4d21-bcfd-c8dcbe1d42f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:48.474752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0d1261-e4a3-4d21-bcfd-c8dcbe1d42f2 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:48.475025 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0d1261-e4a3-4d21-bcfd-c8dcbe1d42f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:48.475293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0d1261-e4a3-4d21-bcfd-c8dcbe1d42f2 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:48.475939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0d1261-e4a3-4d21-bcfd-c8dcbe1d42f2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:48.477360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0d1261-e4a3-4d21-bcfd-c8dcbe1d42f2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:48.498304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 593/1177] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:43:48 2022] POST /identity/v3/users => generated 311 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:43:50.661920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd5230-4e0a-49a1-ad55-af04d6b1d95c None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:50.662482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd5230-4e0a-49a1-ad55-af04d6b1d95c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:50.662482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cd5230-4e0a-49a1-ad55-af04d6b1d95c None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:50.663811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 585/1178] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:50.669365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823fa60d-f840-4e06-b20b-58be6edaedca None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:50.669781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823fa60d-f840-4e06-b20b-58be6edaedca None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:50.669889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823fa60d-f840-4e06-b20b-58be6edaedca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:50.688592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-823fa60d-f840-4e06-b20b-58be6edaedca None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:50.689763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-823fa60d-f840-4e06-b20b-58be6edaedca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:50.700074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:50.754949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 594/1179] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:51.020108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd5230-4e0a-49a1-ad55-af04d6b1d95c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:51.036676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:51.119677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cd5230-4e0a-49a1-ad55-af04d6b1d95c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:51.152629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fd652c-46e7-4560-adc8-7cb85123a013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:51.153243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fd652c-46e7-4560-adc8-7cb85123a013 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:51.153524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fd652c-46e7-4560-adc8-7cb85123a013 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:51.153799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fd652c-46e7-4560-adc8-7cb85123a013 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:51.164449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fd652c-46e7-4560-adc8-7cb85123a013 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-b0fd652c-46e7-4560-adc8-7cb85123a013 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:51.169280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:51.170291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:43:51.171245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 586/1180] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:43:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:51.177067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823fa60d-f840-4e06-b20b-58be6edaedca None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:51.271065 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823fa60d-f840-4e06-b20b-58be6edaedca None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:51.304021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6188dea-f871-473d-a332-8bc03035d850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:51.304603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6188dea-f871-473d-a332-8bc03035d850 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:51.304879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6188dea-f871-473d-a332-8bc03035d850 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:51.305148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6188dea-f871-473d-a332-8bc03035d850 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:51.305799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6188dea-f871-473d-a332-8bc03035d850 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:51.306122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6188dea-f871-473d-a332-8bc03035d850 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:51.307506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6188dea-f871-473d-a332-8bc03035d850 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:51.315218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 595/1181] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:43:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:51.320876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fd652c-46e7-4560-adc8-7cb85123a013 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:51.394030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:51.418145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fd652c-46e7-4560-adc8-7cb85123a013 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:51.449471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f52936-b1a1-475a-9f94-f03ec1ef5130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:51.450094 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f52936-b1a1-475a-9f94-f03ec1ef5130 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:51.450375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f52936-b1a1-475a-9f94-f03ec1ef5130 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:51.450646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f52936-b1a1-475a-9f94-f03ec1ef5130 None admin] PATH_INFO: `/v3/users/563c3fdaee484060be3d9367dd40c0f2` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:51.463573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f52936-b1a1-475a-9f94-f03ec1ef5130 None admin] RBAC: Authorizing `identity:get_user(user_id=563c3fdaee484060be3d9367dd40c0f2)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:51.464985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f52936-b1a1-475a-9f94-f03ec1ef5130 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:51.466572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 587/1182] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:51 2022] GET /identity/v3/users/563c3fdaee484060be3d9367dd40c0f2 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:51.472981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6188dea-f871-473d-a332-8bc03035d850 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:51.483912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:51.568857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6188dea-f871-473d-a332-8bc03035d850 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:51.601371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c46b50-8c93-4c41-a0a7-9861d783c388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:51.601959 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c46b50-8c93-4c41-a0a7-9861d783c388 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:51.602269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c46b50-8c93-4c41-a0a7-9861d783c388 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:51.602546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c46b50-8c93-4c41-a0a7-9861d783c388 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:51.666200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c46b50-8c93-4c41-a0a7-9861d783c388 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:51.667634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c46b50-8c93-4c41-a0a7-9861d783c388 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:51.677865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:51.762689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 596/1183] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:51.768593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f52936-b1a1-475a-9f94-f03ec1ef5130 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:51.863993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f52936-b1a1-475a-9f94-f03ec1ef5130 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:51.892568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:51.897743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aafd49c-0aea-4aae-b663-59d63ded0ff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:51.898384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aafd49c-0aea-4aae-b663-59d63ded0ff9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:51.898665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aafd49c-0aea-4aae-b663-59d63ded0ff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:51.898938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aafd49c-0aea-4aae-b663-59d63ded0ff9 None admin] PATH_INFO: `/v3/projects/412e7b494fd74c4dbc64ac2f605cc7f6` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:51.900028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aafd49c-0aea-4aae-b663-59d63ded0ff9 None admin] RBAC: Authorizing `identity:get_project(project_id=412e7b494fd74c4dbc64ac2f605cc7f6)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:51.901434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aafd49c-0aea-4aae-b663-59d63ded0ff9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:51.903046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 588/1184] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:51 2022] GET /identity/v3/projects/412e7b494fd74c4dbc64ac2f605cc7f6 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:43:51.908901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c46b50-8c93-4c41-a0a7-9861d783c388 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:52.004915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c46b50-8c93-4c41-a0a7-9861d783c388 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:52.013594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:52.039158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deea556c-19ca-408b-98a9-1c12478f2cc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:52.039758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea556c-19ca-408b-98a9-1c12478f2cc5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:52.040033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea556c-19ca-408b-98a9-1c12478f2cc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:52.040305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deea556c-19ca-408b-98a9-1c12478f2cc5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:52.041048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deea556c-19ca-408b-98a9-1c12478f2cc5 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=412e7b494fd74c4dbc64ac2f605cc7f6, user.id=563c3fdaee484060be3d9367dd40c0f2) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:52.041353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deea556c-19ca-408b-98a9-1c12478f2cc5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:52.042870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deea556c-19ca-408b-98a9-1c12478f2cc5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:52.048763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 597/1185] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:43:51 2022] GET /identity/v3/role_assignments?user.id=563c3fdaee484060be3d9367dd40c0f2&scope.project.id=412e7b494fd74c4dbc64ac2f605cc7f6&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:54.149677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b3d510-45ec-41a4-9c1b-2cde019a2fc5 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:54.150239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b3d510-45ec-41a4-9c1b-2cde019a2fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:54.150239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b3d510-45ec-41a4-9c1b-2cde019a2fc5 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:54.151557 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 589/1186] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:54.156999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15139774-8462-4007-a058-d322a04c0c13 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:54.157416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15139774-8462-4007-a058-d322a04c0c13 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:54.157551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15139774-8462-4007-a058-d322a04c0c13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:54.175165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-15139774-8462-4007-a058-d322a04c0c13 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:54.176348 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-15139774-8462-4007-a058-d322a04c0c13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:54.239781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 598/1187] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:54.510016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b3d510-45ec-41a4-9c1b-2cde019a2fc5 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:54.605464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b3d510-45ec-41a4-9c1b-2cde019a2fc5 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:54.637964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c9ca4-2e23-439b-a617-01f0136cfec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:54.638586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c9ca4-2e23-439b-a617-01f0136cfec4 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:54.638858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c9ca4-2e23-439b-a617-01f0136cfec4 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:54.639125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49c9ca4-2e23-439b-a617-01f0136cfec4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:54.648605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49c9ca4-2e23-439b-a617-01f0136cfec4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-d49c9ca4-2e23-439b-a617-01f0136cfec4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:54.653276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:54.654381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:43:54.655222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 590/1188] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:43:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:54.660245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15139774-8462-4007-a058-d322a04c0c13 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:54.754751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15139774-8462-4007-a058-d322a04c0c13 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:54.756212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:54.789015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e9f58e-eceb-4d45-82b7-0725b631fb7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:54.789548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e9f58e-eceb-4d45-82b7-0725b631fb7e None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:54.789824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e9f58e-eceb-4d45-82b7-0725b631fb7e None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:54.790123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e9f58e-eceb-4d45-82b7-0725b631fb7e None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:54.790774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e9f58e-eceb-4d45-82b7-0725b631fb7e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:54.791073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e9f58e-eceb-4d45-82b7-0725b631fb7e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:54.792426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e9f58e-eceb-4d45-82b7-0725b631fb7e None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:54.800729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 599/1189] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:43:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:54.806376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c9ca4-2e23-439b-a617-01f0136cfec4 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:54.863038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:54.901060 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49c9ca4-2e23-439b-a617-01f0136cfec4 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:54.933718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daad7d72-7612-40cf-bb7e-577b3f7a394a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:54.934341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad7d72-7612-40cf-bb7e-577b3f7a394a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:54.934614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad7d72-7612-40cf-bb7e-577b3f7a394a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:54.934879 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad7d72-7612-40cf-bb7e-577b3f7a394a None admin] PATH_INFO: `/v3/users/563c3fdaee484060be3d9367dd40c0f2` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:54.935968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daad7d72-7612-40cf-bb7e-577b3f7a394a None admin] RBAC: Authorizing `identity:get_user(user_id=563c3fdaee484060be3d9367dd40c0f2)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:54.937605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daad7d72-7612-40cf-bb7e-577b3f7a394a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:54.939147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 591/1190] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:54 2022] GET /identity/v3/users/563c3fdaee484060be3d9367dd40c0f2 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:54.944652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e9f58e-eceb-4d45-82b7-0725b631fb7e None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:55.042569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e9f58e-eceb-4d45-82b7-0725b631fb7e None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:55.075928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1698b8ca-0ea5-4759-8197-c9dfd4e61275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:55.076542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1698b8ca-0ea5-4759-8197-c9dfd4e61275 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:55.076820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1698b8ca-0ea5-4759-8197-c9dfd4e61275 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:55.077092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1698b8ca-0ea5-4759-8197-c9dfd4e61275 None admin] PATH_INFO: `/v3/projects/412e7b494fd74c4dbc64ac2f605cc7f6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:55.078265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1698b8ca-0ea5-4759-8197-c9dfd4e61275 None admin] RBAC: Authorizing `identity:get_project(project_id=412e7b494fd74c4dbc64ac2f605cc7f6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:55.079685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1698b8ca-0ea5-4759-8197-c9dfd4e61275 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:55.081287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 600/1191] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:54 2022] GET /identity/v3/projects/412e7b494fd74c4dbc64ac2f605cc7f6 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:55.087316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daad7d72-7612-40cf-bb7e-577b3f7a394a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:55.202783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daad7d72-7612-40cf-bb7e-577b3f7a394a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:55.228328 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:55.236650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efdb9d6-090e-4214-82d8-6d49c38bae6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:55.237292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efdb9d6-090e-4214-82d8-6d49c38bae6f None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:55.237569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efdb9d6-090e-4214-82d8-6d49c38bae6f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:55.237842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efdb9d6-090e-4214-82d8-6d49c38bae6f None admin] PATH_INFO: `/v3/projects/412e7b494fd74c4dbc64ac2f605cc7f6/users/563c3fdaee484060be3d9367dd40c0f2/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:55.239730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efdb9d6-090e-4214-82d8-6d49c38bae6f None admin] RBAC: Authorizing `identity:create_grant(project_id=412e7b494fd74c4dbc64ac2f605cc7f6, user_id=563c3fdaee484060be3d9367dd40c0f2, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:55.241139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efdb9d6-090e-4214-82d8-6d49c38bae6f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:55.254126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 592/1192] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:43:55 2022] PUT /identity/v3/projects/412e7b494fd74c4dbc64ac2f605cc7f6/users/563c3fdaee484060be3d9367dd40c0f2/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:43:57.340110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56bb276-d3a6-4d93-92cc-fe275da0644a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:57.340599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56bb276-d3a6-4d93-92cc-fe275da0644a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:57.340599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56bb276-d3a6-4d93-92cc-fe275da0644a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:57.341934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 601/1193] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:43:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:43:57.347616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2e836-663e-4365-94b2-15809dfb261f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:57.348054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2e836-663e-4365-94b2-15809dfb261f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:57.348148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2e836-663e-4365-94b2-15809dfb261f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:57.365714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8ab2e836-663e-4365-94b2-15809dfb261f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:43:57.366911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8ab2e836-663e-4365-94b2-15809dfb261f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:43:57.429972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 593/1194] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:43:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:43:57.695686 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56bb276-d3a6-4d93-92cc-fe275da0644a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:57.726178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:57.793996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56bb276-d3a6-4d93-92cc-fe275da0644a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:57.826578 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050293f6-5160-435f-ab80-e7ed6ccc7eba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:57.827179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050293f6-5160-435f-ab80-e7ed6ccc7eba None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:57.827450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050293f6-5160-435f-ab80-e7ed6ccc7eba None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:57.827717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-050293f6-5160-435f-ab80-e7ed6ccc7eba None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:57.837820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-050293f6-5160-435f-ab80-e7ed6ccc7eba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-050293f6-5160-435f-ab80-e7ed6ccc7eba None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:43:57.842514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:43:57.844001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:43:57.844936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 602/1195] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:43:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:43:57.850219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab2e836-663e-4365-94b2-15809dfb261f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:57.943527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab2e836-663e-4365-94b2-15809dfb261f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:57.959588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:57.977581 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af9b468-c86d-4960-949a-6f0dbc911571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:57.978196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af9b468-c86d-4960-949a-6f0dbc911571 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:57.978470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af9b468-c86d-4960-949a-6f0dbc911571 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:57.978738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af9b468-c86d-4960-949a-6f0dbc911571 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:57.979381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af9b468-c86d-4960-949a-6f0dbc911571 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:57.979677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af9b468-c86d-4960-949a-6f0dbc911571 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:57.981011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af9b468-c86d-4960-949a-6f0dbc911571 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:57.988567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 594/1196] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:43:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:57.994311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050293f6-5160-435f-ab80-e7ed6ccc7eba None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:58.084131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:58.092838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-050293f6-5160-435f-ab80-e7ed6ccc7eba None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:58.125046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c91b0d8-6ab7-4051-b3e2-d3603c679697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:58.125657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c91b0d8-6ab7-4051-b3e2-d3603c679697 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:58.125932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c91b0d8-6ab7-4051-b3e2-d3603c679697 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:58.126231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c91b0d8-6ab7-4051-b3e2-d3603c679697 None admin] PATH_INFO: `/v3/users/563c3fdaee484060be3d9367dd40c0f2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:58.127334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c91b0d8-6ab7-4051-b3e2-d3603c679697 None admin] RBAC: Authorizing `identity:get_user(user_id=563c3fdaee484060be3d9367dd40c0f2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:58.128749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c91b0d8-6ab7-4051-b3e2-d3603c679697 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:58.130583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 603/1197] 10.222.0.25 () {56 vars in 1177 bytes} [Sun Oct 2 00:43:57 2022] GET /identity/v3/users/563c3fdaee484060be3d9367dd40c0f2 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:58.136473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af9b468-c86d-4960-949a-6f0dbc911571 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:58.233170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af9b468-c86d-4960-949a-6f0dbc911571 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:58.265956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a93cf3-1f60-40a8-8a6a-bc3427d2247b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:58.266572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a93cf3-1f60-40a8-8a6a-bc3427d2247b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:58.266850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a93cf3-1f60-40a8-8a6a-bc3427d2247b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:58.267121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a93cf3-1f60-40a8-8a6a-bc3427d2247b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:58.312788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:58.331334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a93cf3-1f60-40a8-8a6a-bc3427d2247b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:58.332759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a93cf3-1f60-40a8-8a6a-bc3427d2247b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:58.425767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 595/1198] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:43:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:43:58.431697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c91b0d8-6ab7-4051-b3e2-d3603c679697 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:58.534466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c91b0d8-6ab7-4051-b3e2-d3603c679697 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:58.567808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8a1950-0109-49a8-8db7-5fd0df95c8f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:58.568435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a1950-0109-49a8-8db7-5fd0df95c8f5 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:58.568713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a1950-0109-49a8-8db7-5fd0df95c8f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:58.568987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a1950-0109-49a8-8db7-5fd0df95c8f5 None admin] PATH_INFO: `/v3/projects/412e7b494fd74c4dbc64ac2f605cc7f6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:58.570125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8a1950-0109-49a8-8db7-5fd0df95c8f5 None admin] RBAC: Authorizing `identity:get_project(project_id=412e7b494fd74c4dbc64ac2f605cc7f6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:58.571548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8a1950-0109-49a8-8db7-5fd0df95c8f5 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:58.573166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 604/1199] 10.222.0.25 () {56 vars in 1186 bytes} [Sun Oct 2 00:43:58 2022] GET /identity/v3/projects/412e7b494fd74c4dbc64ac2f605cc7f6 => generated 361 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:43:58.579396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a93cf3-1f60-40a8-8a6a-bc3427d2247b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:43:58.658621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:43:58.677841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a93cf3-1f60-40a8-8a6a-bc3427d2247b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:43:58.709536 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0dcea7-08d5-47b8-9c03-f254904b195d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:43:58.710178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0dcea7-08d5-47b8-9c03-f254904b195d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:43:58.710451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0dcea7-08d5-47b8-9c03-f254904b195d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:43:58.710717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0dcea7-08d5-47b8-9c03-f254904b195d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:43:58.711425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0dcea7-08d5-47b8-9c03-f254904b195d None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=412e7b494fd74c4dbc64ac2f605cc7f6, user.id=563c3fdaee484060be3d9367dd40c0f2) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:43:58.711727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0dcea7-08d5-47b8-9c03-f254904b195d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:43:58.713135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0dcea7-08d5-47b8-9c03-f254904b195d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:43:58.719191 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 596/1200] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:43:58 2022] GET /identity/v3/role_assignments?user.id=563c3fdaee484060be3d9367dd40c0f2&scope.project.id=412e7b494fd74c4dbc64ac2f605cc7f6&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:44:08.563577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:44:08.716091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=28 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:44:54.368625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8714e540-c7df-482c-a1ce-51c466b48629 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:54.369181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8714e540-c7df-482c-a1ce-51c466b48629 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:54.369181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8714e540-c7df-482c-a1ce-51c466b48629 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:54.370715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 605/1201] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:44:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:44:54.378049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc0176-fe23-4548-a26b-8f02aa37cf0f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:54.378610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc0176-fe23-4548-a26b-8f02aa37cf0f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:54.378707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dc0176-fe23-4548-a26b-8f02aa37cf0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:54.406197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a6dc0176-fe23-4548-a26b-8f02aa37cf0f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:44:54.407742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a6dc0176-fe23-4548-a26b-8f02aa37cf0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:44:54.503666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 597/1202] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:44:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:44:54.785351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8714e540-c7df-482c-a1ce-51c466b48629 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:54.905899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8714e540-c7df-482c-a1ce-51c466b48629 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:54.950591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17c7ab7-a8c0-47fd-a404-07bdab5ac375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:54.951215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17c7ab7-a8c0-47fd-a404-07bdab5ac375 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:54.951488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17c7ab7-a8c0-47fd-a404-07bdab5ac375 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:54.951817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17c7ab7-a8c0-47fd-a404-07bdab5ac375 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:54.964855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17c7ab7-a8c0-47fd-a404-07bdab5ac375 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:44:54.966589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17c7ab7-a8c0-47fd-a404-07bdab5ac375 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-e17c7ab7-a8c0-47fd-a404-07bdab5ac375 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:44:54.981438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:44:54.982981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:44:54.984263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:44:54.984263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:44:54.984263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:44:54.984263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 606/1203] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:44:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:44:54.989675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dc0176-fe23-4548-a26b-8f02aa37cf0f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:55.125624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dc0176-fe23-4548-a26b-8f02aa37cf0f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:55.161056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:44:55.170765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52d058-a987-403c-97c5-7eb09623e347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:55.171512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52d058-a987-403c-97c5-7eb09623e347 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:55.171819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52d058-a987-403c-97c5-7eb09623e347 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:55.172054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52d058-a987-403c-97c5-7eb09623e347 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:55.173298 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52d058-a987-403c-97c5-7eb09623e347 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:44:55.174039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52d058-a987-403c-97c5-7eb09623e347 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:44:55.175865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52d058-a987-403c-97c5-7eb09623e347 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:44:55.189939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 598/1204] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:44:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:44:55.196171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17c7ab7-a8c0-47fd-a404-07bdab5ac375 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:55.278920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:44:55.339992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17c7ab7-a8c0-47fd-a404-07bdab5ac375 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:55.387479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70430700-1c68-43d3-8537-c056ebc27ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:55.388103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70430700-1c68-43d3-8537-c056ebc27ff4 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:55.388277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70430700-1c68-43d3-8537-c056ebc27ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:55.388551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70430700-1c68-43d3-8537-c056ebc27ff4 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:55.389258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70430700-1c68-43d3-8537-c056ebc27ff4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:44:55.390811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70430700-1c68-43d3-8537-c056ebc27ff4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:44:55.421815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 607/1205] 10.222.0.25 () {60 vars in 1132 bytes} [Sun Oct 2 00:44:55 2022] POST /identity/v3/users => generated 251 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:44:57.738736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bac1b0c-50df-4bf1-9b20-3a78f520d6ab None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:57.738736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bac1b0c-50df-4bf1-9b20-3a78f520d6ab None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:57.739716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bac1b0c-50df-4bf1-9b20-3a78f520d6ab None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:57.741094 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 599/1206] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:44:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:44:57.747201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f22444-7741-4b15-b2c8-068a5d68375c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:57.747634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f22444-7741-4b15-b2c8-068a5d68375c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:57.747916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f22444-7741-4b15-b2c8-068a5d68375c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:57.769306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b5f22444-7741-4b15-b2c8-068a5d68375c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:44:57.770584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b5f22444-7741-4b15-b2c8-068a5d68375c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:44:57.856645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 608/1207] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:44:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:44:58.172508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bac1b0c-50df-4bf1-9b20-3a78f520d6ab None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:58.300830 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bac1b0c-50df-4bf1-9b20-3a78f520d6ab None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:58.339019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338bfca4-2324-49fa-8a89-3c7dec258723 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:58.339654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338bfca4-2324-49fa-8a89-3c7dec258723 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:58.340018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338bfca4-2324-49fa-8a89-3c7dec258723 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:58.340390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338bfca4-2324-49fa-8a89-3c7dec258723 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:58.353148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338bfca4-2324-49fa-8a89-3c7dec258723 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-338bfca4-2324-49fa-8a89-3c7dec258723 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:44:58.359477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:44:58.360546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:44:58.361503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 600/1208] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:44:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:44:58.368901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f22444-7741-4b15-b2c8-068a5d68375c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:58.422731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:44:58.489876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f22444-7741-4b15-b2c8-068a5d68375c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:58.528002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b871fceb-f0a5-4ba9-88f1-0d8d9a58e9cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:58.528608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b871fceb-f0a5-4ba9-88f1-0d8d9a58e9cb None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:58.528884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b871fceb-f0a5-4ba9-88f1-0d8d9a58e9cb None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:58.529155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b871fceb-f0a5-4ba9-88f1-0d8d9a58e9cb None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:58.529845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b871fceb-f0a5-4ba9-88f1-0d8d9a58e9cb None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:44:58.530175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b871fceb-f0a5-4ba9-88f1-0d8d9a58e9cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:44:58.531535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b871fceb-f0a5-4ba9-88f1-0d8d9a58e9cb None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:44:58.539579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 609/1209] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:44:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:44:58.545431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338bfca4-2324-49fa-8a89-3c7dec258723 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:58.604770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:44:58.661985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338bfca4-2324-49fa-8a89-3c7dec258723 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:58.703848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea491746-0502-4d2b-ad67-25eeb3410899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:58.704844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea491746-0502-4d2b-ad67-25eeb3410899 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:58.705387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea491746-0502-4d2b-ad67-25eeb3410899 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:58.705847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea491746-0502-4d2b-ad67-25eeb3410899 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:58.720150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea491746-0502-4d2b-ad67-25eeb3410899 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:44:58.729938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea491746-0502-4d2b-ad67-25eeb3410899 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-ea491746-0502-4d2b-ad67-25eeb3410899 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:44:58.748704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:44:58.753286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:44:58.755020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:44:58.755020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:44:58.755020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:44:58.755020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 601/1210] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:44:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:44:58.760375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b871fceb-f0a5-4ba9-88f1-0d8d9a58e9cb None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:58.901853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:44:58.905901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b871fceb-f0a5-4ba9-88f1-0d8d9a58e9cb None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:58.943509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64d8adf-90ed-46cd-8500-3cc337bb679d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:58.944198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64d8adf-90ed-46cd-8500-3cc337bb679d None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:58.944552 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64d8adf-90ed-46cd-8500-3cc337bb679d None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:58.944933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64d8adf-90ed-46cd-8500-3cc337bb679d None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:58.945622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64d8adf-90ed-46cd-8500-3cc337bb679d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:44:58.946014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64d8adf-90ed-46cd-8500-3cc337bb679d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:44:58.947630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64d8adf-90ed-46cd-8500-3cc337bb679d None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:44:58.962851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 610/1211] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:44:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:44:58.983130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea491746-0502-4d2b-ad67-25eeb3410899 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:59.098038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea491746-0502-4d2b-ad67-25eeb3410899 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:59.099809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:44:59.138572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7be6cde-5cdf-4d9d-a240-865b598b7eee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:59.139902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7be6cde-5cdf-4d9d-a240-865b598b7eee None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:59.140477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7be6cde-5cdf-4d9d-a240-865b598b7eee None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:59.141251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7be6cde-5cdf-4d9d-a240-865b598b7eee None admin] PATH_INFO: `/v3/users/placement` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:59.149810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7be6cde-5cdf-4d9d-a240-865b598b7eee None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:44:59.151619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7be6cde-5cdf-4d9d-a240-865b598b7eee None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-a7be6cde-5cdf-4d9d-a240-865b598b7eee None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:44:59.163615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:44:59.164619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:44:59.165637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 602/1212] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:44:58 2022] GET /identity/v3/users/placement => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:44:59.171921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64d8adf-90ed-46cd-8500-3cc337bb679d None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:59.275257 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64d8adf-90ed-46cd-8500-3cc337bb679d None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:59.314203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752d5776-73a8-40e4-8da1-ede91f591675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:59.314802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752d5776-73a8-40e4-8da1-ede91f591675 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:59.315075 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752d5776-73a8-40e4-8da1-ede91f591675 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:59.315342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752d5776-73a8-40e4-8da1-ede91f591675 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:59.316073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752d5776-73a8-40e4-8da1-ede91f591675 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:44:59.316365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752d5776-73a8-40e4-8da1-ede91f591675 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:44:59.317773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752d5776-73a8-40e4-8da1-ede91f591675 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:44:59.340580 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 611/1213] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Oct 2 00:44:59 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:44:59.341806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7be6cde-5cdf-4d9d-a240-865b598b7eee None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:59.455443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7be6cde-5cdf-4d9d-a240-865b598b7eee None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:59.497820 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babf414d-42e7-4bed-86be-806db9d8f95e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:59.498490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babf414d-42e7-4bed-86be-806db9d8f95e None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:59.498807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babf414d-42e7-4bed-86be-806db9d8f95e None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:59.499063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babf414d-42e7-4bed-86be-806db9d8f95e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:59.532875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:44:59.578015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-babf414d-42e7-4bed-86be-806db9d8f95e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:44:59.579869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-babf414d-42e7-4bed-86be-806db9d8f95e None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:44:59.685996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 603/1214] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:44:59 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:44:59.692120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752d5776-73a8-40e4-8da1-ede91f591675 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:59.716186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:44:59.810758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752d5776-73a8-40e4-8da1-ede91f591675 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:44:59.847728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2e7f3c-507e-4ace-9ce6-08dc60c96878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:44:59.848379 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2e7f3c-507e-4ace-9ce6-08dc60c96878 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:44:59.848729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2e7f3c-507e-4ace-9ce6-08dc60c96878 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:44:59.848937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2e7f3c-507e-4ace-9ce6-08dc60c96878 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:44:59.860818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2e7f3c-507e-4ace-9ce6-08dc60c96878 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:44:59.862518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2e7f3c-507e-4ace-9ce6-08dc60c96878 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-ee2e7f3c-507e-4ace-9ce6-08dc60c96878 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:44:59.880805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:44:59.882193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:44:59.883363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:44:59.883363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:44:59.883363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:44:59.883905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 612/1215] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:44:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:44:59.889767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babf414d-42e7-4bed-86be-806db9d8f95e None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:44:59.957048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:00.001307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babf414d-42e7-4bed-86be-806db9d8f95e None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:00.038124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33720ca-3f90-44cd-82f8-7f24372e69af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:00.038749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33720ca-3f90-44cd-82f8-7f24372e69af None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:00.039030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33720ca-3f90-44cd-82f8-7f24372e69af None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:00.039300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33720ca-3f90-44cd-82f8-7f24372e69af None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:00.039973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33720ca-3f90-44cd-82f8-7f24372e69af None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:00.040306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33720ca-3f90-44cd-82f8-7f24372e69af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:00.041672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33720ca-3f90-44cd-82f8-7f24372e69af None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:00.054579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 604/1216] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:44:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:45:00.060358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2e7f3c-507e-4ace-9ce6-08dc60c96878 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:00.135574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:00.176210 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2e7f3c-507e-4ace-9ce6-08dc60c96878 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:00.219971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b634e663-6b7f-47c8-8d2c-22ff528de58f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:00.220570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b634e663-6b7f-47c8-8d2c-22ff528de58f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:00.220849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b634e663-6b7f-47c8-8d2c-22ff528de58f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:00.221112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b634e663-6b7f-47c8-8d2c-22ff528de58f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:00.230473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b634e663-6b7f-47c8-8d2c-22ff528de58f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:00.232040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b634e663-6b7f-47c8-8d2c-22ff528de58f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-b634e663-6b7f-47c8-8d2c-22ff528de58f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:45:00.245268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:00.246937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:00.248193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:45:00.248193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:45:00.248193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:45:00.248193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:45:00.248193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:45:00.248193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:45:00.248193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:00.248193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:00.248193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 613/1217] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:45:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:00.253373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33720ca-3f90-44cd-82f8-7f24372e69af None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:00.364195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33720ca-3f90-44cd-82f8-7f24372e69af None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:00.389572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:00.401477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27256c6-ae08-418f-8108-1c09e0064a52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:00.402099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27256c6-ae08-418f-8108-1c09e0064a52 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:00.402422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27256c6-ae08-418f-8108-1c09e0064a52 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:00.402695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27256c6-ae08-418f-8108-1c09e0064a52 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:00.403376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27256c6-ae08-418f-8108-1c09e0064a52 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:00.403677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27256c6-ae08-418f-8108-1c09e0064a52 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:00.405085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27256c6-ae08-418f-8108-1c09e0064a52 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:00.416665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 605/1218] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:45:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:00.423805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b634e663-6b7f-47c8-8d2c-22ff528de58f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:00.528175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b634e663-6b7f-47c8-8d2c-22ff528de58f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:00.557537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:00.565753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a41eac8-ae5e-4764-8ec6-3e31c0c79f6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:00.566494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41eac8-ae5e-4764-8ec6-3e31c0c79f6b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:00.566870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41eac8-ae5e-4764-8ec6-3e31c0c79f6b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:00.567172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a41eac8-ae5e-4764-8ec6-3e31c0c79f6b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:00.567958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a41eac8-ae5e-4764-8ec6-3e31c0c79f6b None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=5eeff4f989ae4338a182bfdadd3a8b63) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:00.568325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a41eac8-ae5e-4764-8ec6-3e31c0c79f6b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:00.569769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a41eac8-ae5e-4764-8ec6-3e31c0c79f6b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:00.576723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 614/1219] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:45:00 2022] GET /identity/v3/role_assignments?user.id=5eeff4f989ae4338a182bfdadd3a8b63&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:02.954491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aa6105-e34d-4647-9ca8-7764a8f3e8c3 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:02.955121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aa6105-e34d-4647-9ca8-7764a8f3e8c3 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:02.955121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aa6105-e34d-4647-9ca8-7764a8f3e8c3 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:02.956730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 606/1220] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:45:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:45:02.962174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addcc2a4-dc00-48d2-aca6-77dccf23b338 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:02.962598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addcc2a4-dc00-48d2-aca6-77dccf23b338 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:02.962798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addcc2a4-dc00-48d2-aca6-77dccf23b338 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:02.983306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-addcc2a4-dc00-48d2-aca6-77dccf23b338 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:45:02.984688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-addcc2a4-dc00-48d2-aca6-77dccf23b338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:45:03.059046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 615/1221] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:45:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:45:03.354311 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aa6105-e34d-4647-9ca8-7764a8f3e8c3 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:03.463302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aa6105-e34d-4647-9ca8-7764a8f3e8c3 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:03.505440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa9afa1-00d9-4496-b006-fa78202a3e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:03.506258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa9afa1-00d9-4496-b006-fa78202a3e2c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:03.506615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa9afa1-00d9-4496-b006-fa78202a3e2c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:03.506904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa9afa1-00d9-4496-b006-fa78202a3e2c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:03.517665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa9afa1-00d9-4496-b006-fa78202a3e2c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-9fa9afa1-00d9-4496-b006-fa78202a3e2c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:45:03.522423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:03.523553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:03.524749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 607/1222] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:45:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:03.530303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addcc2a4-dc00-48d2-aca6-77dccf23b338 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:03.636114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addcc2a4-dc00-48d2-aca6-77dccf23b338 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:03.661913 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:03.674994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac721bf6-d0fa-4bf4-a949-d2045efebb90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:03.675598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac721bf6-d0fa-4bf4-a949-d2045efebb90 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:03.675877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac721bf6-d0fa-4bf4-a949-d2045efebb90 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:03.676145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac721bf6-d0fa-4bf4-a949-d2045efebb90 None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:03.676794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac721bf6-d0fa-4bf4-a949-d2045efebb90 None admin] RBAC: Adding query filter params (name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:03.677092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac721bf6-d0fa-4bf4-a949-d2045efebb90 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:03.678625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac721bf6-d0fa-4bf4-a949-d2045efebb90 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:03.687599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 616/1223] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:45:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:03.694293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa9afa1-00d9-4496-b006-fa78202a3e2c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:03.740792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:03.801932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa9afa1-00d9-4496-b006-fa78202a3e2c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:03.841812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400088aa-3140-4ff3-9bbf-661f4b566732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:03.842461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400088aa-3140-4ff3-9bbf-661f4b566732 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:03.842734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400088aa-3140-4ff3-9bbf-661f4b566732 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:03.843002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400088aa-3140-4ff3-9bbf-661f4b566732 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:03.853250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400088aa-3140-4ff3-9bbf-661f4b566732 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:03.854707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400088aa-3140-4ff3-9bbf-661f4b566732 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-400088aa-3140-4ff3-9bbf-661f4b566732 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:45:03.869300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:03.870342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:45:03.871796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:45:03.871796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:03.871796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:03.871796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 608/1224] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:03.876754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac721bf6-d0fa-4bf4-a949-d2045efebb90 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:03.983944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac721bf6-d0fa-4bf4-a949-d2045efebb90 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:04.016669 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b40de4-c0b3-410a-906d-9ce30763d17f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:04.017267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b40de4-c0b3-410a-906d-9ce30763d17f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:04.017547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b40de4-c0b3-410a-906d-9ce30763d17f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:04.017802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b40de4-c0b3-410a-906d-9ce30763d17f None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:04.018462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b40de4-c0b3-410a-906d-9ce30763d17f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:04.018755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b40de4-c0b3-410a-906d-9ce30763d17f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:04.020112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b40de4-c0b3-410a-906d-9ce30763d17f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:04.036852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 617/1225] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:45:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:04.042396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400088aa-3140-4ff3-9bbf-661f4b566732 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:04.144713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:04.152048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400088aa-3140-4ff3-9bbf-661f4b566732 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:04.186091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2416ed2c-68d5-43dd-88cb-b0249a86e70b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:04.186708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2416ed2c-68d5-43dd-88cb-b0249a86e70b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:04.186976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2416ed2c-68d5-43dd-88cb-b0249a86e70b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:04.187249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2416ed2c-68d5-43dd-88cb-b0249a86e70b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:04.194585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2416ed2c-68d5-43dd-88cb-b0249a86e70b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:04.195974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2416ed2c-68d5-43dd-88cb-b0249a86e70b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-2416ed2c-68d5-43dd-88cb-b0249a86e70b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:45:04.206713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:04.207934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:04.209219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 609/1226] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:04 2022] GET /identity/v3/users/placement => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:04.213791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b40de4-c0b3-410a-906d-9ce30763d17f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:04.261004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:04.318846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b40de4-c0b3-410a-906d-9ce30763d17f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:04.352950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10813615-d598-441d-884e-a35d95298cc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:04.353592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10813615-d598-441d-884e-a35d95298cc9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:04.353904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10813615-d598-441d-884e-a35d95298cc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:04.354248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10813615-d598-441d-884e-a35d95298cc9 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:04.355515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10813615-d598-441d-884e-a35d95298cc9 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:04.355806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10813615-d598-441d-884e-a35d95298cc9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:04.357352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10813615-d598-441d-884e-a35d95298cc9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:04.376197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 618/1227] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Oct 2 00:45:04 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:45:04.382268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2416ed2c-68d5-43dd-88cb-b0249a86e70b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:04.485636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2416ed2c-68d5-43dd-88cb-b0249a86e70b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:04.521604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2394f75f-e74a-4845-9618-c0ae2c1be873 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:04.522185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394f75f-e74a-4845-9618-c0ae2c1be873 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:04.522459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394f75f-e74a-4845-9618-c0ae2c1be873 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:04.522725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394f75f-e74a-4845-9618-c0ae2c1be873 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:04.533745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2394f75f-e74a-4845-9618-c0ae2c1be873 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:04.535190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2394f75f-e74a-4845-9618-c0ae2c1be873 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-2394f75f-e74a-4845-9618-c0ae2c1be873 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:45:04.549132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:04.550610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:45:04.551809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:45:04.551809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:04.551809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:04.551809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 610/1228] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:04.556082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10813615-d598-441d-884e-a35d95298cc9 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:04.663168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10813615-d598-441d-884e-a35d95298cc9 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:04.665485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:04.704670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6c3175-998e-49fd-89bc-7fe8e1e20ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:04.705395 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6c3175-998e-49fd-89bc-7fe8e1e20ee3 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:04.705858 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6c3175-998e-49fd-89bc-7fe8e1e20ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:04.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6c3175-998e-49fd-89bc-7fe8e1e20ee3 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:04.707037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6c3175-998e-49fd-89bc-7fe8e1e20ee3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:04.707420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6c3175-998e-49fd-89bc-7fe8e1e20ee3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:04.709496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6c3175-998e-49fd-89bc-7fe8e1e20ee3 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:04.729055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 619/1229] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:45:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:04.730932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2394f75f-e74a-4845-9618-c0ae2c1be873 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:04.740234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:04.840179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2394f75f-e74a-4845-9618-c0ae2c1be873 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:04.881138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314eb4ce-e626-40f9-9598-679642464326 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:04.881767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314eb4ce-e626-40f9-9598-679642464326 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:04.882045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314eb4ce-e626-40f9-9598-679642464326 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:04.882344 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314eb4ce-e626-40f9-9598-679642464326 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:04.890985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314eb4ce-e626-40f9-9598-679642464326 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:04.892439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314eb4ce-e626-40f9-9598-679642464326 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-314eb4ce-e626-40f9-9598-679642464326 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:45:04.904816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:04.905815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:04.906774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:45:04.906774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:45:04.906774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:45:04.906774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:45:04.906774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:45:04.906774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:45:04.906774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:04.906774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:04.906774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 611/1230] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:45:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:04.912455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6c3175-998e-49fd-89bc-7fe8e1e20ee3 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:05.029388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6c3175-998e-49fd-89bc-7fe8e1e20ee3 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:05.066899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431a4e53-bbd4-43f7-a950-c37dab8f5846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:05.067520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431a4e53-bbd4-43f7-a950-c37dab8f5846 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:05.067822 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431a4e53-bbd4-43f7-a950-c37dab8f5846 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:05.068143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431a4e53-bbd4-43f7-a950-c37dab8f5846 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:05.068908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431a4e53-bbd4-43f7-a950-c37dab8f5846 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:05.069284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431a4e53-bbd4-43f7-a950-c37dab8f5846 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:05.070826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431a4e53-bbd4-43f7-a950-c37dab8f5846 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:05.083631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 620/1231] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:45:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:05.090470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314eb4ce-e626-40f9-9598-679642464326 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:05.155480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:05.192603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314eb4ce-e626-40f9-9598-679642464326 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:05.228461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec48c1b-c506-4b6c-87f4-7e4435352e65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:05.229199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec48c1b-c506-4b6c-87f4-7e4435352e65 None admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:05.229478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec48c1b-c506-4b6c-87f4-7e4435352e65 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:05.229750 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec48c1b-c506-4b6c-87f4-7e4435352e65 None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/5eeff4f989ae4338a182bfdadd3a8b63/roles/2c1bcaa088914d4593bb1debe2fc88d6` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:05.246528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec48c1b-c506-4b6c-87f4-7e4435352e65 None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=5eeff4f989ae4338a182bfdadd3a8b63, role_id=2c1bcaa088914d4593bb1debe2fc88d6)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:05.248178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec48c1b-c506-4b6c-87f4-7e4435352e65 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:05.261128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 612/1232] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:45:05 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/5eeff4f989ae4338a182bfdadd3a8b63/roles/2c1bcaa088914d4593bb1debe2fc88d6 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:45:07.394997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d264f6f0-4954-436b-bd26-282e6ba633b4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:07.395541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d264f6f0-4954-436b-bd26-282e6ba633b4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:07.395541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d264f6f0-4954-436b-bd26-282e6ba633b4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:07.397096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 621/1233] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:45:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:45:07.403585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17939147-3877-4980-9ecb-5bf1da5632d6 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:07.404124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17939147-3877-4980-9ecb-5bf1da5632d6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:07.404124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17939147-3877-4980-9ecb-5bf1da5632d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:07.424888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-17939147-3877-4980-9ecb-5bf1da5632d6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:45:07.426366 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-17939147-3877-4980-9ecb-5bf1da5632d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:45:07.503043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 613/1234] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:45:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:45:07.804187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d264f6f0-4954-436b-bd26-282e6ba633b4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:07.834706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:07.923183 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d264f6f0-4954-436b-bd26-282e6ba633b4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:07.960641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bc7664-8cf1-409d-aaff-787bff6705ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:07.961328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bc7664-8cf1-409d-aaff-787bff6705ea None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:07.961604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bc7664-8cf1-409d-aaff-787bff6705ea None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:07.961873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bc7664-8cf1-409d-aaff-787bff6705ea None admin] PATH_INFO: `/v3/roles/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:07.973949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bc7664-8cf1-409d-aaff-787bff6705ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-86bc7664-8cf1-409d-aaff-787bff6705ea None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:45:07.979105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:07.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:07.983367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 622/1235] 10.222.0.25 () {56 vars in 1102 bytes} [Sun Oct 2 00:45:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:07.986475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17939147-3877-4980-9ecb-5bf1da5632d6 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:08.039419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:08.100645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17939147-3877-4980-9ecb-5bf1da5632d6 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:08.139877 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4aa6bc-b3ae-4764-891b-cbd6f403ba7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:08.140508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4aa6bc-b3ae-4764-891b-cbd6f403ba7b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:08.140921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4aa6bc-b3ae-4764-891b-cbd6f403ba7b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:08.141221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4aa6bc-b3ae-4764-891b-cbd6f403ba7b None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:08.141840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4aa6bc-b3ae-4764-891b-cbd6f403ba7b None admin] RBAC: Adding query filter params (name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:08.142162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4aa6bc-b3ae-4764-891b-cbd6f403ba7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:08.143498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4aa6bc-b3ae-4764-891b-cbd6f403ba7b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:08.152811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 614/1236] 10.222.0.25 () {56 vars in 1103 bytes} [Sun Oct 2 00:45:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:08.159055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bc7664-8cf1-409d-aaff-787bff6705ea None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:08.253961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:08.272612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bc7664-8cf1-409d-aaff-787bff6705ea None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:08.311475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c9f050-4467-4294-ac24-7f4e780273e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:08.312092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c9f050-4467-4294-ac24-7f4e780273e8 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:08.312363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c9f050-4467-4294-ac24-7f4e780273e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:08.312628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c9f050-4467-4294-ac24-7f4e780273e8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:08.323396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c9f050-4467-4294-ac24-7f4e780273e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:08.325398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c9f050-4467-4294-ac24-7f4e780273e8 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-15c9f050-4467-4294-ac24-7f4e780273e8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:45:08.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:08.343340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:45:08.344761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:45:08.344761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:08.344761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:08.344761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 623/1237] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 02 00:45:08.351378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4aa6bc-b3ae-4764-891b-cbd6f403ba7b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:08.459099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4aa6bc-b3ae-4764-891b-cbd6f403ba7b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:08.470296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:08.499833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f71f20-92d3-4ee3-a0b8-e9b6cbbd3848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:08.500432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f71f20-92d3-4ee3-a0b8-e9b6cbbd3848 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:08.500706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f71f20-92d3-4ee3-a0b8-e9b6cbbd3848 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:08.500976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f71f20-92d3-4ee3-a0b8-e9b6cbbd3848 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:08.501601 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f71f20-92d3-4ee3-a0b8-e9b6cbbd3848 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:08.502033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f71f20-92d3-4ee3-a0b8-e9b6cbbd3848 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:08.503821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f71f20-92d3-4ee3-a0b8-e9b6cbbd3848 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:08.515154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 615/1238] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:45:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:45:08.521625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c9f050-4467-4294-ac24-7f4e780273e8 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:08.639416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c9f050-4467-4294-ac24-7f4e780273e8 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:08.685303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89814148-38de-4da8-9f6e-bbbbcd78c93f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:08.686483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89814148-38de-4da8-9f6e-bbbbcd78c93f None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:08.686788 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89814148-38de-4da8-9f6e-bbbbcd78c93f None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:08.687197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89814148-38de-4da8-9f6e-bbbbcd78c93f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:08.691068 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:08.699156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89814148-38de-4da8-9f6e-bbbbcd78c93f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:08.700967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89814148-38de-4da8-9f6e-bbbbcd78c93f None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-89814148-38de-4da8-9f6e-bbbbcd78c93f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:45:08.717592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:08.719544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:08.720802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 624/1239] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:08 2022] GET /identity/v3/users/placement => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:08.725017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f71f20-92d3-4ee3-a0b8-e9b6cbbd3848 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:08.843874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f71f20-92d3-4ee3-a0b8-e9b6cbbd3848 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:08.880757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05e7d92-27f3-4640-b7c4-f5296dd8b355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:08.881360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05e7d92-27f3-4640-b7c4-f5296dd8b355 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:08.881639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05e7d92-27f3-4640-b7c4-f5296dd8b355 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:08.881911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05e7d92-27f3-4640-b7c4-f5296dd8b355 None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:08.882663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05e7d92-27f3-4640-b7c4-f5296dd8b355 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:08.882968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05e7d92-27f3-4640-b7c4-f5296dd8b355 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:08.884423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05e7d92-27f3-4640-b7c4-f5296dd8b355 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:08.903040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 616/1240] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Oct 2 00:45:08 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:45:08.909418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89814148-38de-4da8-9f6e-bbbbcd78c93f None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:09.021109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89814148-38de-4da8-9f6e-bbbbcd78c93f None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:09.059950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af5a311-79e9-41b7-8edf-62d9f4154812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:09.060553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af5a311-79e9-41b7-8edf-62d9f4154812 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:09.060827 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af5a311-79e9-41b7-8edf-62d9f4154812 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:09.061094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af5a311-79e9-41b7-8edf-62d9f4154812 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:09.133141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:09.140330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af5a311-79e9-41b7-8edf-62d9f4154812 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:09.141783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af5a311-79e9-41b7-8edf-62d9f4154812 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:09.251741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 625/1241] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:45:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:45:09.258046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05e7d92-27f3-4640-b7c4-f5296dd8b355 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:09.288902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:09.369209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05e7d92-27f3-4640-b7c4-f5296dd8b355 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:09.406213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6c502b-91d2-4cc8-b299-bd667085404c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:09.406928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6c502b-91d2-4cc8-b299-bd667085404c None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:09.407239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6c502b-91d2-4cc8-b299-bd667085404c None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:09.407568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6c502b-91d2-4cc8-b299-bd667085404c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:09.418911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6c502b-91d2-4cc8-b299-bd667085404c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:09.420495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6c502b-91d2-4cc8-b299-bd667085404c None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-9d6c502b-91d2-4cc8-b299-bd667085404c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:45:09.437022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:09.438046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:45:09.438985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:45:09.438985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:09.438985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:09.438985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 617/1242] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:09.444313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af5a311-79e9-41b7-8edf-62d9f4154812 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:09.547749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:09.554702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af5a311-79e9-41b7-8edf-62d9f4154812 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:09.601971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74ba9cf-4fea-4c1b-a394-2fcf7cc01662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:09.602722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74ba9cf-4fea-4c1b-a394-2fcf7cc01662 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:09.603057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74ba9cf-4fea-4c1b-a394-2fcf7cc01662 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:09.603335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74ba9cf-4fea-4c1b-a394-2fcf7cc01662 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:09.603981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74ba9cf-4fea-4c1b-a394-2fcf7cc01662 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:09.604268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74ba9cf-4fea-4c1b-a394-2fcf7cc01662 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:09.606180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74ba9cf-4fea-4c1b-a394-2fcf7cc01662 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:09.619593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 626/1243] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:45:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:09.626345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6c502b-91d2-4cc8-b299-bd667085404c None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:09.720074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:09.745268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6c502b-91d2-4cc8-b299-bd667085404c None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:09.784985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423e8166-9b10-4fc1-9021-ac6945421f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:09.785639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423e8166-9b10-4fc1-9021-ac6945421f2a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:09.785919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423e8166-9b10-4fc1-9021-ac6945421f2a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:09.786227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423e8166-9b10-4fc1-9021-ac6945421f2a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:09.794776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423e8166-9b10-4fc1-9021-ac6945421f2a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:09.796227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423e8166-9b10-4fc1-9021-ac6945421f2a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-423e8166-9b10-4fc1-9021-ac6945421f2a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:45:09.809404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:09.810629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:09.811936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:45:09.811936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:45:09.811936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:45:09.811936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:45:09.811936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:45:09.811936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:45:09.811936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:09.811936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:09.811936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 618/1244] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:45:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:09.817102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74ba9cf-4fea-4c1b-a394-2fcf7cc01662 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:09.925197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74ba9cf-4fea-4c1b-a394-2fcf7cc01662 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:09.960722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3647508-38f9-44e0-91eb-8eb1c2140aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:09.961332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3647508-38f9-44e0-91eb-8eb1c2140aef None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:09.961608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3647508-38f9-44e0-91eb-8eb1c2140aef None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:09.961877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3647508-38f9-44e0-91eb-8eb1c2140aef None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:09.962658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3647508-38f9-44e0-91eb-8eb1c2140aef None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:09.962958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3647508-38f9-44e0-91eb-8eb1c2140aef None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:09.964875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3647508-38f9-44e0-91eb-8eb1c2140aef None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:09.976572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 627/1245] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:45:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:09.982932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423e8166-9b10-4fc1-9021-ac6945421f2a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:10.099051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423e8166-9b10-4fc1-9021-ac6945421f2a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:10.137317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6893d99d-ee6d-4d57-9b24-67184b6a5147 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:10.138008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6893d99d-ee6d-4d57-9b24-67184b6a5147 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:10.138379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6893d99d-ee6d-4d57-9b24-67184b6a5147 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:10.138679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6893d99d-ee6d-4d57-9b24-67184b6a5147 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:10.139420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6893d99d-ee6d-4d57-9b24-67184b6a5147 None admin] RBAC: Adding query filter params (role.id=2c1bcaa088914d4593bb1debe2fc88d6, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=5eeff4f989ae4338a182bfdadd3a8b63) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:10.139765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6893d99d-ee6d-4d57-9b24-67184b6a5147 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:10.141569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6893d99d-ee6d-4d57-9b24-67184b6a5147 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:10.148235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 619/1246] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:45:09 2022] GET /identity/v3/role_assignments?user.id=5eeff4f989ae4338a182bfdadd3a8b63&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=2c1bcaa088914d4593bb1debe2fc88d6 => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:12.482817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28487d8b-1c08-476e-bfe2-972235345255 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:12.483350 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28487d8b-1c08-476e-bfe2-972235345255 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:12.483350 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28487d8b-1c08-476e-bfe2-972235345255 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:12.484614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 628/1247] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:45:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:45:12.490289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5c40b0-2e08-4951-8bee-1cbd27df2772 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:12.490711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5c40b0-2e08-4951-8bee-1cbd27df2772 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:12.490812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5c40b0-2e08-4951-8bee-1cbd27df2772 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:12.497462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:12.511945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-af5c40b0-2e08-4951-8bee-1cbd27df2772 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:45:12.513118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-af5c40b0-2e08-4951-8bee-1cbd27df2772 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:45:12.590641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 620/1248] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:45:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:45:12.859396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28487d8b-1c08-476e-bfe2-972235345255 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:12.878620 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:12.970525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28487d8b-1c08-476e-bfe2-972235345255 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:13.008192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614c5e2a-70ec-4d84-b922-a906d1af3331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:13.008832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614c5e2a-70ec-4d84-b922-a906d1af3331 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:13.009109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614c5e2a-70ec-4d84-b922-a906d1af3331 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:13.009376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614c5e2a-70ec-4d84-b922-a906d1af3331 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:13.020613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614c5e2a-70ec-4d84-b922-a906d1af3331 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-614c5e2a-70ec-4d84-b922-a906d1af3331 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:45:13.025614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:13.027443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:45:13.028334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:13.030504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 629/1249] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:45:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:13.033545 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5c40b0-2e08-4951-8bee-1cbd27df2772 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:13.135253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5c40b0-2e08-4951-8bee-1cbd27df2772 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:13.165764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:13.171528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff575fc-4225-4f1b-a236-51f311fa9925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:13.172145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff575fc-4225-4f1b-a236-51f311fa9925 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:13.172416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff575fc-4225-4f1b-a236-51f311fa9925 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:13.172682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff575fc-4225-4f1b-a236-51f311fa9925 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:13.173348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff575fc-4225-4f1b-a236-51f311fa9925 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:13.173645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff575fc-4225-4f1b-a236-51f311fa9925 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:13.175049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff575fc-4225-4f1b-a236-51f311fa9925 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:13.184015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 621/1250] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:45:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:13.190181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614c5e2a-70ec-4d84-b922-a906d1af3331 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:13.269589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:13.293892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614c5e2a-70ec-4d84-b922-a906d1af3331 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:13.330089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1915c1d2-2f61-4ebe-84e8-5b4eb73a1f62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:13.330726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915c1d2-2f61-4ebe-84e8-5b4eb73a1f62 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:13.331000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915c1d2-2f61-4ebe-84e8-5b4eb73a1f62 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:13.331269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915c1d2-2f61-4ebe-84e8-5b4eb73a1f62 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:13.341493 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1915c1d2-2f61-4ebe-84e8-5b4eb73a1f62 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:13.343015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1915c1d2-2f61-4ebe-84e8-5b4eb73a1f62 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1915c1d2-2f61-4ebe-84e8-5b4eb73a1f62 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:45:13.357916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:13.359074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:45:13.360198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:45:13.360198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:13.360198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:13.360198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 630/1251] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:13.366319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff575fc-4225-4f1b-a236-51f311fa9925 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:13.467213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff575fc-4225-4f1b-a236-51f311fa9925 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:13.500626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da2bec1-5fb2-4f2c-870e-8113d1397d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:13.501225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da2bec1-5fb2-4f2c-870e-8113d1397d9f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:13.501496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da2bec1-5fb2-4f2c-870e-8113d1397d9f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:13.501878 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da2bec1-5fb2-4f2c-870e-8113d1397d9f None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:13.502813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da2bec1-5fb2-4f2c-870e-8113d1397d9f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:13.503112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da2bec1-5fb2-4f2c-870e-8113d1397d9f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:13.504471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da2bec1-5fb2-4f2c-870e-8113d1397d9f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:13.516784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 622/1252] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:45:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:13.522668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1915c1d2-2f61-4ebe-84e8-5b4eb73a1f62 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:13.619934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1915c1d2-2f61-4ebe-84e8-5b4eb73a1f62 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:13.645716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:13.653903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0cd681-c71c-45ca-9e77-c4fab9ea3482 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:13.654545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0cd681-c71c-45ca-9e77-c4fab9ea3482 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:13.654824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0cd681-c71c-45ca-9e77-c4fab9ea3482 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:13.655096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0cd681-c71c-45ca-9e77-c4fab9ea3482 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:13.662917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0cd681-c71c-45ca-9e77-c4fab9ea3482 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:13.664355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0cd681-c71c-45ca-9e77-c4fab9ea3482 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-be0cd681-c71c-45ca-9e77-c4fab9ea3482 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:45:13.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:13.677557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:13.678576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 631/1253] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:13 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:13.684308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da2bec1-5fb2-4f2c-870e-8113d1397d9f None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:13.743571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:13.786796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da2bec1-5fb2-4f2c-870e-8113d1397d9f None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:13.819377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f429f43c-02da-448a-94e4-f99e8055ff1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:13.819967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f429f43c-02da-448a-94e4-f99e8055ff1b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:13.820243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f429f43c-02da-448a-94e4-f99e8055ff1b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:13.820509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f429f43c-02da-448a-94e4-f99e8055ff1b None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:13.821212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f429f43c-02da-448a-94e4-f99e8055ff1b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:13.821513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f429f43c-02da-448a-94e4-f99e8055ff1b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:13.822944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f429f43c-02da-448a-94e4-f99e8055ff1b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:13.839962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 623/1254] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Oct 2 00:45:13 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:13.845852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0cd681-c71c-45ca-9e77-c4fab9ea3482 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:13.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0cd681-c71c-45ca-9e77-c4fab9ea3482 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:13.974155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e4a21d-4c62-40f0-b7d2-3534c6d42655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:13.974732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e4a21d-4c62-40f0-b7d2-3534c6d42655 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:13.975005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e4a21d-4c62-40f0-b7d2-3534c6d42655 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:13.975273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e4a21d-4c62-40f0-b7d2-3534c6d42655 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:14.028474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:14.041010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e4a21d-4c62-40f0-b7d2-3534c6d42655 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:14.042992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e4a21d-4c62-40f0-b7d2-3534c6d42655 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:14.137737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 632/1255] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:45:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:45:14.143648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f429f43c-02da-448a-94e4-f99e8055ff1b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:14.239333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f429f43c-02da-448a-94e4-f99e8055ff1b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:14.251740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:14.272238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4452f97-97fd-4588-acc0-a60cc4267d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:14.272854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4452f97-97fd-4588-acc0-a60cc4267d2d None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:14.273133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4452f97-97fd-4588-acc0-a60cc4267d2d None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:14.273405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4452f97-97fd-4588-acc0-a60cc4267d2d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:14.283565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4452f97-97fd-4588-acc0-a60cc4267d2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:14.284932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4452f97-97fd-4588-acc0-a60cc4267d2d None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-e4452f97-97fd-4588-acc0-a60cc4267d2d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:45:14.299443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:14.300549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:45:14.301669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:45:14.301669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:14.301669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:14.301669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 624/1256] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:14.306213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e4a21d-4c62-40f0-b7d2-3534c6d42655 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:14.388588 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:14.400304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e4a21d-4c62-40f0-b7d2-3534c6d42655 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:14.431702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f4ce26-9132-417d-ad1c-7bab44128013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:14.432286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4ce26-9132-417d-ad1c-7bab44128013 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:14.433086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4ce26-9132-417d-ad1c-7bab44128013 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:14.433352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f4ce26-9132-417d-ad1c-7bab44128013 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:14.433968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f4ce26-9132-417d-ad1c-7bab44128013 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:14.434295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f4ce26-9132-417d-ad1c-7bab44128013 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:14.435637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f4ce26-9132-417d-ad1c-7bab44128013 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:14.446113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 633/1257] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:45:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:14.452194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4452f97-97fd-4588-acc0-a60cc4267d2d None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:14.548510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4452f97-97fd-4588-acc0-a60cc4267d2d None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:14.580362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db078e66-55ab-4afb-b80f-595859626758 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:14.580965 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db078e66-55ab-4afb-b80f-595859626758 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:14.581243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db078e66-55ab-4afb-b80f-595859626758 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:14.581512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db078e66-55ab-4afb-b80f-595859626758 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:14.587861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db078e66-55ab-4afb-b80f-595859626758 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:14.589354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db078e66-55ab-4afb-b80f-595859626758 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-db078e66-55ab-4afb-b80f-595859626758 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:45:14.600189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:14.601246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:14.602374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:45:14.602374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:45:14.602374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:45:14.602374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:45:14.602374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:45:14.602374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:45:14.602374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:14.602374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:14.602374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 625/1258] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:45:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:14.607489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f4ce26-9132-417d-ad1c-7bab44128013 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:14.715794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f4ce26-9132-417d-ad1c-7bab44128013 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:14.747661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce4741c-559e-476d-94d2-8c66df158765 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:14.748252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4741c-559e-476d-94d2-8c66df158765 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:14.748527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4741c-559e-476d-94d2-8c66df158765 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:14.748795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce4741c-559e-476d-94d2-8c66df158765 None admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:14.749470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce4741c-559e-476d-94d2-8c66df158765 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:14.749770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce4741c-559e-476d-94d2-8c66df158765 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:14.751215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce4741c-559e-476d-94d2-8c66df158765 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:14.763313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 634/1259] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:45:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:14.769513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db078e66-55ab-4afb-b80f-595859626758 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:14.794681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:14.866496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db078e66-55ab-4afb-b80f-595859626758 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:14.898348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061857a0-f6ca-4f2a-9939-409f3bcabb27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:14.898932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061857a0-f6ca-4f2a-9939-409f3bcabb27 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:14.899205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061857a0-f6ca-4f2a-9939-409f3bcabb27 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:14.899473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061857a0-f6ca-4f2a-9939-409f3bcabb27 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:14.900173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061857a0-f6ca-4f2a-9939-409f3bcabb27 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=5eeff4f989ae4338a182bfdadd3a8b63) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:14.900474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061857a0-f6ca-4f2a-9939-409f3bcabb27 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:14.901904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061857a0-f6ca-4f2a-9939-409f3bcabb27 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:14.906976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 626/1260] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:45:14 2022] GET /identity/v3/role_assignments?user.id=5eeff4f989ae4338a182bfdadd3a8b63&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:17.012621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292e96ef-becf-4bc7-97f4-f4c3735a68aa None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:17.013172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292e96ef-becf-4bc7-97f4-f4c3735a68aa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:17.013172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292e96ef-becf-4bc7-97f4-f4c3735a68aa None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:17.014505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 635/1261] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:45:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:45:17.019565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ecabf0-25f3-49b9-ba0e-fa08c2c4ba51 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:17.019953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ecabf0-25f3-49b9-ba0e-fa08c2c4ba51 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:17.020102 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ecabf0-25f3-49b9-ba0e-fa08c2c4ba51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:17.041838 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-18ecabf0-25f3-49b9-ba0e-fa08c2c4ba51 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:45:17.043100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-18ecabf0-25f3-49b9-ba0e-fa08c2c4ba51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:45:17.108113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 627/1262] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:45:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:45:17.372713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292e96ef-becf-4bc7-97f4-f4c3735a68aa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:17.383556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:17.471220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292e96ef-becf-4bc7-97f4-f4c3735a68aa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:17.504237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc51c10c-ecf5-434f-abe1-d5e06bda9374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:17.504836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc51c10c-ecf5-434f-abe1-d5e06bda9374 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:17.505112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc51c10c-ecf5-434f-abe1-d5e06bda9374 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:17.505380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc51c10c-ecf5-434f-abe1-d5e06bda9374 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:17.515022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc51c10c-ecf5-434f-abe1-d5e06bda9374 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-bc51c10c-ecf5-434f-abe1-d5e06bda9374 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:45:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:17.520779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:17.521798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 636/1263] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:45:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:17.527147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ecabf0-25f3-49b9-ba0e-fa08c2c4ba51 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:17.545242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:17.622050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ecabf0-25f3-49b9-ba0e-fa08c2c4ba51 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:17.654936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb7fdf-c17a-476b-850e-265e11702697 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:17.655473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb7fdf-c17a-476b-850e-265e11702697 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:17.655750 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb7fdf-c17a-476b-850e-265e11702697 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:17.656022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb7fdf-c17a-476b-850e-265e11702697 None admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:17.656674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb7fdf-c17a-476b-850e-265e11702697 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:17.656975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb7fdf-c17a-476b-850e-265e11702697 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:17.658343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb7fdf-c17a-476b-850e-265e11702697 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:17.665996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 628/1264] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:45:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:17.672232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc51c10c-ecf5-434f-abe1-d5e06bda9374 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:17.746484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:17.780169 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc51c10c-ecf5-434f-abe1-d5e06bda9374 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:17.813123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65adeb1-2e11-4299-9fe6-a6f9d7c481ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:17.813737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65adeb1-2e11-4299-9fe6-a6f9d7c481ae None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:17.814011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65adeb1-2e11-4299-9fe6-a6f9d7c481ae None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:17.814319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65adeb1-2e11-4299-9fe6-a6f9d7c481ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:17.823808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65adeb1-2e11-4299-9fe6-a6f9d7c481ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:17.825240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65adeb1-2e11-4299-9fe6-a6f9d7c481ae None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-a65adeb1-2e11-4299-9fe6-a6f9d7c481ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:45:17.839114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:17.840305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:45:17.841251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:45:17.841251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:17.841251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:17.841251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 637/1265] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:17.845989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb7fdf-c17a-476b-850e-265e11702697 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:17.923716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:17.942598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb7fdf-c17a-476b-850e-265e11702697 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:17.974396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b8fd51-2e7c-46ae-a120-8759ae448033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:17.974983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b8fd51-2e7c-46ae-a120-8759ae448033 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:17.975260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b8fd51-2e7c-46ae-a120-8759ae448033 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:17.975528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b8fd51-2e7c-46ae-a120-8759ae448033 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:17.976152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b8fd51-2e7c-46ae-a120-8759ae448033 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:17.976451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b8fd51-2e7c-46ae-a120-8759ae448033 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:17.977786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b8fd51-2e7c-46ae-a120-8759ae448033 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:17.988761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 629/1266] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:45:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:17.994363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65adeb1-2e11-4299-9fe6-a6f9d7c481ae None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:18.091981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65adeb1-2e11-4299-9fe6-a6f9d7c481ae None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:18.111054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:18.124369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c77d573-6e72-493c-850f-ab548429abaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:18.124981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c77d573-6e72-493c-850f-ab548429abaf None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:18.125254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c77d573-6e72-493c-850f-ab548429abaf None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:18.125518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c77d573-6e72-493c-850f-ab548429abaf None admin] PATH_INFO: `/v3/users/placement` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:18.132752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c77d573-6e72-493c-850f-ab548429abaf None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:18.134207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c77d573-6e72-493c-850f-ab548429abaf None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-2c77d573-6e72-493c-850f-ab548429abaf None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:45:18.144890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:18.145980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:18.147256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 638/1267] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:17 2022] GET /identity/v3/users/placement => generated 87 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:18.152281 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b8fd51-2e7c-46ae-a120-8759ae448033 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:18.246344 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b8fd51-2e7c-46ae-a120-8759ae448033 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:18.278620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcf0898-5476-4223-812f-5913afb37f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:18.279204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcf0898-5476-4223-812f-5913afb37f6f None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:18.279477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcf0898-5476-4223-812f-5913afb37f6f None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:18.279745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcf0898-5476-4223-812f-5913afb37f6f None admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:18.280445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcf0898-5476-4223-812f-5913afb37f6f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:18.280744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcf0898-5476-4223-812f-5913afb37f6f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:18.282170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcf0898-5476-4223-812f-5913afb37f6f None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:18.290346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:18.299046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 630/1268] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Oct 2 00:45:18 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:18.304616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c77d573-6e72-493c-850f-ab548429abaf None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:18.403447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c77d573-6e72-493c-850f-ab548429abaf None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:18.434891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b93e69f-ac9c-463f-bf1f-039c42602b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:18.435497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93e69f-ac9c-463f-bf1f-039c42602b17 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:18.435772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93e69f-ac9c-463f-bf1f-039c42602b17 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:18.436041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b93e69f-ac9c-463f-bf1f-039c42602b17 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:18.446108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b93e69f-ac9c-463f-bf1f-039c42602b17 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:18.447534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b93e69f-ac9c-463f-bf1f-039c42602b17 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-7b93e69f-ac9c-463f-bf1f-039c42602b17 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:45:18.461630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:18.462775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:45:18.464376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:45:18.464376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:18.464376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:18.464376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 639/1269] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:18.469047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcf0898-5476-4223-812f-5913afb37f6f None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:18.563188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcf0898-5476-4223-812f-5913afb37f6f None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:18.595349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7842b61a-e9c6-4ff9-b4e1-073424813811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:18.595930 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7842b61a-e9c6-4ff9-b4e1-073424813811 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:18.596205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7842b61a-e9c6-4ff9-b4e1-073424813811 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:18.596474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7842b61a-e9c6-4ff9-b4e1-073424813811 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:18.597099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7842b61a-e9c6-4ff9-b4e1-073424813811 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:18.597394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7842b61a-e9c6-4ff9-b4e1-073424813811 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:18.598757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7842b61a-e9c6-4ff9-b4e1-073424813811 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:18.609839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 631/1270] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:45:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:18.615389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b93e69f-ac9c-463f-bf1f-039c42602b17 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:18.646053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:18.719808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b93e69f-ac9c-463f-bf1f-039c42602b17 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:18.751755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d78e260-203e-4e6e-ac8b-cd2697df9852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:18.752629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d78e260-203e-4e6e-ac8b-cd2697df9852 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:18.752901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d78e260-203e-4e6e-ac8b-cd2697df9852 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:18.753168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d78e260-203e-4e6e-ac8b-cd2697df9852 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:18.759397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d78e260-203e-4e6e-ac8b-cd2697df9852 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:18.760813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d78e260-203e-4e6e-ac8b-cd2697df9852 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-0d78e260-203e-4e6e-ac8b-cd2697df9852 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:45:18.771528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:18.772567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:18.773704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:45:18.773704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:45:18.773704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:45:18.773704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:45:18.773704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:45:18.773704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:45:18.773704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:18.773704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:18.773704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 640/1271] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:45:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:18.779076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7842b61a-e9c6-4ff9-b4e1-073424813811 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:18.833861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:18.875032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7842b61a-e9c6-4ff9-b4e1-073424813811 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:18.906795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a948ddc-de94-4e08-914b-9cf2888c9b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:18.907378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a948ddc-de94-4e08-914b-9cf2888c9b50 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:18.907649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a948ddc-de94-4e08-914b-9cf2888c9b50 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:18.907917 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a948ddc-de94-4e08-914b-9cf2888c9b50 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:18.908589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a948ddc-de94-4e08-914b-9cf2888c9b50 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:18.908886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a948ddc-de94-4e08-914b-9cf2888c9b50 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:18.910308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a948ddc-de94-4e08-914b-9cf2888c9b50 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:18.922234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 632/1272] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:45:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:18.928311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d78e260-203e-4e6e-ac8b-cd2697df9852 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:19.015281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:19.024231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d78e260-203e-4e6e-ac8b-cd2697df9852 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:19.057816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552109b6-bfac-4174-adff-48f15ef960fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:19.058490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552109b6-bfac-4174-adff-48f15ef960fe None admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:19.058766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552109b6-bfac-4174-adff-48f15ef960fe None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:19.059036 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552109b6-bfac-4174-adff-48f15ef960fe None admin] PATH_INFO: `/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/5eeff4f989ae4338a182bfdadd3a8b63/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:19.060884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552109b6-bfac-4174-adff-48f15ef960fe None admin] RBAC: Authorizing `identity:create_grant(project_id=8204e43e3cf9467cb423bb0d0f87c4fe, user_id=5eeff4f989ae4338a182bfdadd3a8b63, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:19.062328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552109b6-bfac-4174-adff-48f15ef960fe None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:19.075113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 641/1273] 10.222.0.25 () {58 vars in 1439 bytes} [Sun Oct 2 00:45:18 2022] PUT /identity/v3/projects/8204e43e3cf9467cb423bb0d0f87c4fe/users/5eeff4f989ae4338a182bfdadd3a8b63/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:45:21.178408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcce10e9-2798-4c4c-9517-dffeb8c41ff7 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:21.178947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcce10e9-2798-4c4c-9517-dffeb8c41ff7 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:21.178947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcce10e9-2798-4c4c-9517-dffeb8c41ff7 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:21.180241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 633/1274] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:45:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:45:21.185313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b00e40-1f54-4715-81d6-c922745e0768 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:21.185703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b00e40-1f54-4715-81d6-c922745e0768 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:21.185890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b00e40-1f54-4715-81d6-c922745e0768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:21.203566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a3b00e40-1f54-4715-81d6-c922745e0768 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:45:21.204745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a3b00e40-1f54-4715-81d6-c922745e0768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:45:21.268629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 642/1275] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:45:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:45:21.535291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcce10e9-2798-4c4c-9517-dffeb8c41ff7 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:21.632266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcce10e9-2798-4c4c-9517-dffeb8c41ff7 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:21.648260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:21.664359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7834e4e7-8ef8-4747-977b-c44a8a09f838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:21.664967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7834e4e7-8ef8-4747-977b-c44a8a09f838 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:21.665243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7834e4e7-8ef8-4747-977b-c44a8a09f838 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:21.665514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7834e4e7-8ef8-4747-977b-c44a8a09f838 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:21.676227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7834e4e7-8ef8-4747-977b-c44a8a09f838 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-7834e4e7-8ef8-4747-977b-c44a8a09f838 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 02 00:45:21.680987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:21.682042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:21.683812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 634/1276] 10.222.0.25 () {56 vars in 1096 bytes} [Sun Oct 2 00:45:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:21.689239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b00e40-1f54-4715-81d6-c922745e0768 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:21.780080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:21.798709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b00e40-1f54-4715-81d6-c922745e0768 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:21.831548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b939bd-b8db-4d64-b0b5-fc00362f12bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:21.832143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b939bd-b8db-4d64-b0b5-fc00362f12bd None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:21.832418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b939bd-b8db-4d64-b0b5-fc00362f12bd None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:21.832688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b939bd-b8db-4d64-b0b5-fc00362f12bd None admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:21.833335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b939bd-b8db-4d64-b0b5-fc00362f12bd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:21.833632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b939bd-b8db-4d64-b0b5-fc00362f12bd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:21.835008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b939bd-b8db-4d64-b0b5-fc00362f12bd None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:21.843318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 643/1277] 10.222.0.25 () {56 vars in 1099 bytes} [Sun Oct 2 00:45:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:45:21.849102 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7834e4e7-8ef8-4747-977b-c44a8a09f838 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:21.944844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7834e4e7-8ef8-4747-977b-c44a8a09f838 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:21.976981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32387ccc-4fc7-4ce3-9b70-1a24d4081854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:21.977581 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32387ccc-4fc7-4ce3-9b70-1a24d4081854 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:21.977858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32387ccc-4fc7-4ce3-9b70-1a24d4081854 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:21.978155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32387ccc-4fc7-4ce3-9b70-1a24d4081854 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:21.987750 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32387ccc-4fc7-4ce3-9b70-1a24d4081854 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:21.989172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32387ccc-4fc7-4ce3-9b70-1a24d4081854 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-32387ccc-4fc7-4ce3-9b70-1a24d4081854 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:45:22.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:22.004954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:45:22.006251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:45:22.006251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:22.006251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:22.006251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 635/1278] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:22.011184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b939bd-b8db-4d64-b0b5-fc00362f12bd None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:22.108679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b939bd-b8db-4d64-b0b5-fc00362f12bd None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:22.140617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcd9545-75ed-409d-9d80-5f15b615d842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:22.141208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcd9545-75ed-409d-9d80-5f15b615d842 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:22.141482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcd9545-75ed-409d-9d80-5f15b615d842 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:22.141752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbcd9545-75ed-409d-9d80-5f15b615d842 None admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:22.142403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcd9545-75ed-409d-9d80-5f15b615d842 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:22.142703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcd9545-75ed-409d-9d80-5f15b615d842 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:22.144055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbcd9545-75ed-409d-9d80-5f15b615d842 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:22.155107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 644/1279] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:45:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:22.160765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32387ccc-4fc7-4ce3-9b70-1a24d4081854 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:22.191276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:22.258006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32387ccc-4fc7-4ce3-9b70-1a24d4081854 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:22.290762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69ec839-5f94-4513-99eb-f7e429328dc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:22.291370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69ec839-5f94-4513-99eb-f7e429328dc9 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:22.291684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69ec839-5f94-4513-99eb-f7e429328dc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:22.291957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69ec839-5f94-4513-99eb-f7e429328dc9 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:22.299052 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69ec839-5f94-4513-99eb-f7e429328dc9 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:22.300468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69ec839-5f94-4513-99eb-f7e429328dc9 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-d69ec839-5f94-4513-99eb-f7e429328dc9 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 02 00:45:22.311241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:22.312176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:22.313239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 636/1280] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:22 2022] GET /identity/v3/users/placement => generated 87 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:22.317998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcd9545-75ed-409d-9d80-5f15b615d842 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:22.321670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:22.415477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbcd9545-75ed-409d-9d80-5f15b615d842 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:22.450133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beea2ee7-7dc0-4ddd-a6ea-a9a374520a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:22.450724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beea2ee7-7dc0-4ddd-a6ea-a9a374520a28 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:22.451001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beea2ee7-7dc0-4ddd-a6ea-a9a374520a28 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:22.451275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beea2ee7-7dc0-4ddd-a6ea-a9a374520a28 None admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:22.451982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beea2ee7-7dc0-4ddd-a6ea-a9a374520a28 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:22.452283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beea2ee7-7dc0-4ddd-a6ea-a9a374520a28 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:22.453689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beea2ee7-7dc0-4ddd-a6ea-a9a374520a28 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:22.469818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 645/1281] 10.222.0.25 () {56 vars in 1143 bytes} [Sun Oct 2 00:45:22 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:22.475606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69ec839-5f94-4513-99eb-f7e429328dc9 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:22.562545 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:22.572081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69ec839-5f94-4513-99eb-f7e429328dc9 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:22.604884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa40ad2-fc39-47c0-badd-644f86d2cc9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:22.605463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa40ad2-fc39-47c0-badd-644f86d2cc9b None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:22.605740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa40ad2-fc39-47c0-badd-644f86d2cc9b None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:22.606010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa40ad2-fc39-47c0-badd-644f86d2cc9b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:22.669837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa40ad2-fc39-47c0-badd-644f86d2cc9b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:22.671692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa40ad2-fc39-47c0-badd-644f86d2cc9b None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:22.755131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:22.765547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 637/1282] 10.222.0.25 () {58 vars in 1282 bytes} [Sun Oct 2 00:45:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 02 00:45:22.771263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beea2ee7-7dc0-4ddd-a6ea-a9a374520a28 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:22.829480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:22.866081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beea2ee7-7dc0-4ddd-a6ea-a9a374520a28 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:22.898378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff37e4d-6dee-4d07-a60e-220db2fd9bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:22.898975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff37e4d-6dee-4d07-a60e-220db2fd9bed None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:22.899246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff37e4d-6dee-4d07-a60e-220db2fd9bed None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:22.899510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff37e4d-6dee-4d07-a60e-220db2fd9bed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:22.909861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff37e4d-6dee-4d07-a60e-220db2fd9bed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:22.911326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff37e4d-6dee-4d07-a60e-220db2fd9bed None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-3ff37e4d-6dee-4d07-a60e-220db2fd9bed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 02 00:45:22.925365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:22.926508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 02 00:45:22.927677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 02 00:45:22.927677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 02 00:45:22.927677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:22.927677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 646/1283] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:45:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:22.932085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa40ad2-fc39-47c0-badd-644f86d2cc9b None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:23.025750 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa40ad2-fc39-47c0-badd-644f86d2cc9b None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:23.060279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409976f8-5eb0-4008-a863-7c91390f8386 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:23.060876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409976f8-5eb0-4008-a863-7c91390f8386 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:23.061151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409976f8-5eb0-4008-a863-7c91390f8386 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:23.061422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409976f8-5eb0-4008-a863-7c91390f8386 None admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:23.062049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409976f8-5eb0-4008-a863-7c91390f8386 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:23.062376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409976f8-5eb0-4008-a863-7c91390f8386 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:23.063722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409976f8-5eb0-4008-a863-7c91390f8386 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:23.074907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 638/1284] 10.222.0.25 () {56 vars in 1109 bytes} [Sun Oct 2 00:45:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:23.080473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff37e4d-6dee-4d07-a60e-220db2fd9bed None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:23.174996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff37e4d-6dee-4d07-a60e-220db2fd9bed None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:23.188410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:23.208618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db6f5b5-9a58-4910-9688-38f1ede2a578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:23.209221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db6f5b5-9a58-4910-9688-38f1ede2a578 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:23.209495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db6f5b5-9a58-4910-9688-38f1ede2a578 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:23.209762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db6f5b5-9a58-4910-9688-38f1ede2a578 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:23.216174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db6f5b5-9a58-4910-9688-38f1ede2a578 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:23.217590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db6f5b5-9a58-4910-9688-38f1ede2a578 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application [None req-1db6f5b5-9a58-4910-9688-38f1ede2a578 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 02 00:45:23.228381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:23.229425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:23.230574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 02 00:45:23.230574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 02 00:45:23.230574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 02 00:45:23.230574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 02 00:45:23.230574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 02 00:45:23.230574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 02 00:45:23.230574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 02 00:45:23.230574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: ERROR keystone.server.flask.application Oct 02 00:45:23.230574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 647/1285] 10.222.0.25 () {56 vars in 1111 bytes} [Sun Oct 2 00:45:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:23.235108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409976f8-5eb0-4008-a863-7c91390f8386 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:23.283233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:23.331220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409976f8-5eb0-4008-a863-7c91390f8386 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:23.362283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0208a501-67e7-4cf1-a8e8-60e453bdbb66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:23.362869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0208a501-67e7-4cf1-a8e8-60e453bdbb66 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:23.363146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0208a501-67e7-4cf1-a8e8-60e453bdbb66 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:23.363418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0208a501-67e7-4cf1-a8e8-60e453bdbb66 None admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:23.364093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0208a501-67e7-4cf1-a8e8-60e453bdbb66 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:23.364394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0208a501-67e7-4cf1-a8e8-60e453bdbb66 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:23.365794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0208a501-67e7-4cf1-a8e8-60e453bdbb66 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:23.377863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 639/1286] 10.222.0.25 () {56 vars in 1148 bytes} [Sun Oct 2 00:45:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:23.383681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db6f5b5-9a58-4910-9688-38f1ede2a578 None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:23.479968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db6f5b5-9a58-4910-9688-38f1ede2a578 None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:23.511744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa98e06-c518-4b8e-a27d-d5a4d00e77f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:23.512330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa98e06-c518-4b8e-a27d-d5a4d00e77f4 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:23.512606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa98e06-c518-4b8e-a27d-d5a4d00e77f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:23.512877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa98e06-c518-4b8e-a27d-d5a4d00e77f4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:23.513585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa98e06-c518-4b8e-a27d-d5a4d00e77f4 None admin] RBAC: Adding query filter params (role.id=e6c41ef419da4c0d98f540b23c2e1a8a, scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe, user.id=5eeff4f989ae4338a182bfdadd3a8b63) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:23.513888 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa98e06-c518-4b8e-a27d-d5a4d00e77f4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:23.515324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa98e06-c518-4b8e-a27d-d5a4d00e77f4 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:23.520826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 648/1287] 10.222.0.25 () {56 vars in 1374 bytes} [Sun Oct 2 00:45:23 2022] GET /identity/v3/role_assignments?user.id=5eeff4f989ae4338a182bfdadd3a8b63&scope.project.id=8204e43e3cf9467cb423bb0d0f87c4fe&role.id=e6c41ef419da4c0d98f540b23c2e1a8a => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:25.664963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eec242-4bdf-4db8-9826-2c21f84c5fc1 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:25.665509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eec242-4bdf-4db8-9826-2c21f84c5fc1 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:25.665509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eec242-4bdf-4db8-9826-2c21f84c5fc1 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:25.667467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 640/1288] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:45:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:45:25.673453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a2b6af-fe4e-4050-b1dc-39a19873206a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:25.673947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a2b6af-fe4e-4050-b1dc-39a19873206a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:25.674309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a2b6af-fe4e-4050-b1dc-39a19873206a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:25.698350 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c2a2b6af-fe4e-4050-b1dc-39a19873206a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:45:25.699697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c2a2b6af-fe4e-4050-b1dc-39a19873206a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:45:25.735337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:25.773451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 649/1289] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:45:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 02 00:45:26.044621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eec242-4bdf-4db8-9826-2c21f84c5fc1 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:26.142276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eec242-4bdf-4db8-9826-2c21f84c5fc1 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:26.152914 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:26.175344 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b809540-4390-4125-ac43-58f2a051c358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:26.175955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b809540-4390-4125-ac43-58f2a051c358 None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:26.176233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b809540-4390-4125-ac43-58f2a051c358 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:26.176506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b809540-4390-4125-ac43-58f2a051c358 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:26.177075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b809540-4390-4125-ac43-58f2a051c358 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:26.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b809540-4390-4125-ac43-58f2a051c358 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-0b809540-4390-4125-ac43-58f2a051c358 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:26.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:26.188781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:26.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 641/1290] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 2 00:45:26 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:26.194452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a2b6af-fe4e-4050-b1dc-39a19873206a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:26.288092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a2b6af-fe4e-4050-b1dc-39a19873206a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:26.319853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859d20aa-4cd2-46e3-b817-5b89cea15ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:26.320440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859d20aa-4cd2-46e3-b817-5b89cea15ee2 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:26.320717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859d20aa-4cd2-46e3-b817-5b89cea15ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:26.320985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859d20aa-4cd2-46e3-b817-5b89cea15ee2 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:26.321596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859d20aa-4cd2-46e3-b817-5b89cea15ee2 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:26.321893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859d20aa-4cd2-46e3-b817-5b89cea15ee2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:26.323994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859d20aa-4cd2-46e3-b817-5b89cea15ee2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:26.328562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 650/1291] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Oct 2 00:45:26 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:26.334085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b809540-4390-4125-ac43-58f2a051c358 None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:26.429512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b809540-4390-4125-ac43-58f2a051c358 None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:26.461871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2902a1aa-5466-4499-b8b4-4366bd20acde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:26.462471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2902a1aa-5466-4499-b8b4-4366bd20acde None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:26.462747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2902a1aa-5466-4499-b8b4-4366bd20acde None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:26.463016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2902a1aa-5466-4499-b8b4-4366bd20acde None admin] PATH_INFO: `/v3/services` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:26.463655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2902a1aa-5466-4499-b8b4-4366bd20acde None admin] RBAC: Adding query filter params (type=placement) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:26.463949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2902a1aa-5466-4499-b8b4-4366bd20acde None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:26.465279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2902a1aa-5466-4499-b8b4-4366bd20acde None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:26.469457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 642/1292] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Oct 2 00:45:26 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:28.556200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a42208-6fa1-4436-b7c4-75a3f0659c4e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:28.557206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a42208-6fa1-4436-b7c4-75a3f0659c4e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:28.557297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a42208-6fa1-4436-b7c4-75a3f0659c4e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:28.558613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 651/1293] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:45:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:45:28.563706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e378f30-9678-43f3-9c39-6bdc09380299 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:28.564124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e378f30-9678-43f3-9c39-6bdc09380299 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:28.564247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e378f30-9678-43f3-9c39-6bdc09380299 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:28.582396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-5e378f30-9678-43f3-9c39-6bdc09380299 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:45:28.583549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5e378f30-9678-43f3-9c39-6bdc09380299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:45:28.610924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:28.647197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 643/1294] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:45:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:45:28.932112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a42208-6fa1-4436-b7c4-75a3f0659c4e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:28.962441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:29.032819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a42208-6fa1-4436-b7c4-75a3f0659c4e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:29.065341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80391e95-ad93-406a-896a-fdb9cd146bc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:29.065935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80391e95-ad93-406a-896a-fdb9cd146bc2 None admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:29.066312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80391e95-ad93-406a-896a-fdb9cd146bc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:29.066586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80391e95-ad93-406a-896a-fdb9cd146bc2 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:29.067237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80391e95-ad93-406a-896a-fdb9cd146bc2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:29.068604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80391e95-ad93-406a-896a-fdb9cd146bc2 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:29.083040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 652/1295] 10.222.0.25 () {60 vars in 1141 bytes} [Sun Oct 2 00:45:28 2022] POST /identity/v3/services => generated 246 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:45:31.206715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c80dd9-40ff-4749-adee-57d778dacf64 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:31.207262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c80dd9-40ff-4749-adee-57d778dacf64 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:31.207262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c80dd9-40ff-4749-adee-57d778dacf64 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:31.208588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 644/1296] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:45:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:45:31.214482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0837cdb-6691-4588-913f-8d715403911b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:31.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0837cdb-6691-4588-913f-8d715403911b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:31.215038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0837cdb-6691-4588-913f-8d715403911b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:31.232962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b0837cdb-6691-4588-913f-8d715403911b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:45:31.234160 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b0837cdb-6691-4588-913f-8d715403911b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:45:31.304303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 653/1297] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:45:31 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:45:31.573951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c80dd9-40ff-4749-adee-57d778dacf64 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:31.671360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c80dd9-40ff-4749-adee-57d778dacf64 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:31.708211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f63708-c15d-4f05-930e-174560377afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:31.708843 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f63708-c15d-4f05-930e-174560377afe None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:31.709153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f63708-c15d-4f05-930e-174560377afe None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:31.709468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f63708-c15d-4f05-930e-174560377afe None admin] PATH_INFO: `/v3/services/placement` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:31.710123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f63708-c15d-4f05-930e-174560377afe None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:31.711630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f63708-c15d-4f05-930e-174560377afe None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-13f63708-c15d-4f05-930e-174560377afe None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:31.721241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:31.722320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:31.723181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 645/1298] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 2 00:45:31 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:31.729026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0837cdb-6691-4588-913f-8d715403911b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:31.752704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:31.832434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0837cdb-6691-4588-913f-8d715403911b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:31.864885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad7c865-fe7a-4d5d-8000-16b57b08307b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:31.865479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad7c865-fe7a-4d5d-8000-16b57b08307b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:31.865755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad7c865-fe7a-4d5d-8000-16b57b08307b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:31.866025 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad7c865-fe7a-4d5d-8000-16b57b08307b None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:31.866671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad7c865-fe7a-4d5d-8000-16b57b08307b None admin] RBAC: Adding query filter params (name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:31.866970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad7c865-fe7a-4d5d-8000-16b57b08307b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:31.868329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad7c865-fe7a-4d5d-8000-16b57b08307b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:31.873352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 654/1299] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Oct 2 00:45:31 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:31.879626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f63708-c15d-4f05-930e-174560377afe None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:31.910921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:31.976273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f63708-c15d-4f05-930e-174560377afe None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:32.007634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3da109-6574-495c-ab77-c9b37466070a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:32.008212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3da109-6574-495c-ab77-c9b37466070a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:32.008484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3da109-6574-495c-ab77-c9b37466070a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:32.008747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3da109-6574-495c-ab77-c9b37466070a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:32.009423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3da109-6574-495c-ab77-c9b37466070a None admin] RBAC: Adding query filter params (interface=public, service_id=e907df9dd6a74e778874b14c01db8c75, region_id=RegionOne) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:32.010289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3da109-6574-495c-ab77-c9b37466070a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:32.011621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3da109-6574-495c-ab77-c9b37466070a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:32.016456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 646/1300] 10.222.0.25 () {56 vars in 1251 bytes} [Sun Oct 2 00:45:31 2022] GET /identity/v3/endpoints?service_id=e907df9dd6a74e778874b14c01db8c75&interface=public®ion_id=RegionOne => generated 194 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:32.021984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad7c865-fe7a-4d5d-8000-16b57b08307b None admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:32.101185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:32.120023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad7c865-fe7a-4d5d-8000-16b57b08307b None admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:32.151869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e01cdce-3996-46c2-8a1d-1e7e7aa7e0b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:32.152461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e01cdce-3996-46c2-8a1d-1e7e7aa7e0b9 None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:32.152739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e01cdce-3996-46c2-8a1d-1e7e7aa7e0b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:32.153012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e01cdce-3996-46c2-8a1d-1e7e7aa7e0b9 None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:32.153609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e01cdce-3996-46c2-8a1d-1e7e7aa7e0b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:32.154996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e01cdce-3996-46c2-8a1d-1e7e7aa7e0b9 None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:32.161658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 655/1301] 10.222.0.25 () {56 vars in 1087 bytes} [Sun Oct 2 00:45:32 2022] GET /identity/v3/services => generated 2191 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:45:34.275529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683f428e-08c5-459c-a9d1-2640a34a0c79 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:34.276063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683f428e-08c5-459c-a9d1-2640a34a0c79 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:34.276063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683f428e-08c5-459c-a9d1-2640a34a0c79 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:34.277396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 647/1302] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:45:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:45:34.282554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac355d4-47a1-4f34-ad6e-d9095a29021d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:34.282963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac355d4-47a1-4f34-ad6e-d9095a29021d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:34.283096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac355d4-47a1-4f34-ad6e-d9095a29021d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:34.301007 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4ac355d4-47a1-4f34-ad6e-d9095a29021d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:45:34.302208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4ac355d4-47a1-4f34-ad6e-d9095a29021d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:45:34.364769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 656/1303] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:45:34 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 02 00:45:34.628736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683f428e-08c5-459c-a9d1-2640a34a0c79 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:34.723068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:34.730661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683f428e-08c5-459c-a9d1-2640a34a0c79 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:34.763537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f713dedb-d928-406b-a1b5-1039e3167c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:34.764148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f713dedb-d928-406b-a1b5-1039e3167c4a None admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:34.764428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f713dedb-d928-406b-a1b5-1039e3167c4a None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:34.764701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f713dedb-d928-406b-a1b5-1039e3167c4a None admin] PATH_INFO: `/v3/services/placement` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:34.765274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f713dedb-d928-406b-a1b5-1039e3167c4a None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:34.766654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f713dedb-d928-406b-a1b5-1039e3167c4a None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application [None req-f713dedb-d928-406b-a1b5-1039e3167c4a None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 02 00:45:34.775128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 02 00:45:34.776307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.get_or_create( Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application with Lock( Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self._enter() Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application return self.creator() Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application created_value = creator( Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: ERROR keystone.server.flask.application Oct 02 00:45:34.777312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 648/1304] 10.222.0.25 () {56 vars in 1117 bytes} [Sun Oct 2 00:45:34 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 02 00:45:34.782179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac355d4-47a1-4f34-ad6e-d9095a29021d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:34.855665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:34.878378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac355d4-47a1-4f34-ad6e-d9095a29021d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:34.910040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e842c8-4958-4ed6-afce-c3b60c7ad93b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:34.910607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e842c8-4958-4ed6-afce-c3b60c7ad93b None admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:34.910881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e842c8-4958-4ed6-afce-c3b60c7ad93b None admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:34.911146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e842c8-4958-4ed6-afce-c3b60c7ad93b None admin] PATH_INFO: `/v3/services` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:34.911754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e842c8-4958-4ed6-afce-c3b60c7ad93b None admin] RBAC: Adding query filter params (name=placement) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:45:34.912050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e842c8-4958-4ed6-afce-c3b60c7ad93b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:34.913398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e842c8-4958-4ed6-afce-c3b60c7ad93b None admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:34.918913 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 657/1305] 10.222.0.25 () {56 vars in 1116 bytes} [Sun Oct 2 00:45:34 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:45:34.924740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f713dedb-d928-406b-a1b5-1039e3167c4a None admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:45:35.021134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f713dedb-d928-406b-a1b5-1039e3167c4a None admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:45:35.063282 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c42fb91-d483-447e-ae51-5ab5e10e7948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:45:35.063878 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c42fb91-d483-447e-ae51-5ab5e10e7948 None admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:45:35.064156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c42fb91-d483-447e-ae51-5ab5e10e7948 None admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:45:35.064427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c42fb91-d483-447e-ae51-5ab5e10e7948 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:45:35.065096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c42fb91-d483-447e-ae51-5ab5e10e7948 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:45:35.066479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c42fb91-d483-447e-ae51-5ab5e10e7948 None admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:45:35.093199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 649/1306] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:45:34 2022] POST /identity/v3/endpoints => generated 331 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:45:44.915939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=30 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:45:45.088140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:39.130761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd15bb-491d-4b5a-95f4-1ee6e6ccfd01 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:39.131291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd15bb-491d-4b5a-95f4-1ee6e6ccfd01 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:39.131291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd15bb-491d-4b5a-95f4-1ee6e6ccfd01 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:39.132846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 658/1307] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:46:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:46:39.141032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06191695-cff2-4918-9698-17b92f48fd88 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:39.142153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06191695-cff2-4918-9698-17b92f48fd88 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:39.142771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06191695-cff2-4918-9698-17b92f48fd88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:39.291423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-06191695-cff2-4918-9698-17b92f48fd88 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:46:39.292778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-06191695-cff2-4918-9698-17b92f48fd88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:39.419149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 650/1308] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:46:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:46:39.710341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e72818-7bf2-42db-9f27-bb033cc6f0b4 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:39.710637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e72818-7bf2-42db-9f27-bb033cc6f0b4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:39.711286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e72818-7bf2-42db-9f27-bb033cc6f0b4 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:39.712951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 659/1309] 10.222.0.25 () {54 vars in 964 bytes} [Sun Oct 2 00:46:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:46:39.719616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c19ff5b-dfeb-4dc9-adf5-6ecfd6f6c9e3 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:39.719953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c19ff5b-dfeb-4dc9-adf5-6ecfd6f6c9e3 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:39.720163 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c19ff5b-dfeb-4dc9-adf5-6ecfd6f6c9e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:39.811254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4c19ff5b-dfeb-4dc9-adf5-6ecfd6f6c9e3 None None] MFA Rules not processed for user `e75811ee9ed7490985fed336ab9feb19`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:46:39.812472 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4c19ff5b-dfeb-4dc9-adf5-6ecfd6f6c9e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:39.834611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:39.895987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 651/1310] 10.222.0.25 () {58 vars in 1061 bytes} [Sun Oct 2 00:46:39 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:46:39.905630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e72818-7bf2-42db-9f27-bb033cc6f0b4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:46:39.907575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-63e72818-7bf2-42db-9f27-bb033cc6f0b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:40.015075 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e72818-7bf2-42db-9f27-bb033cc6f0b4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:46:40.099230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f0117f-add4-4439-a2ed-c158884cc15a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:46:40.099957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f0117f-add4-4439-a2ed-c158884cc15a service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:40.100338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f0117f-add4-4439-a2ed-c158884cc15a service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:40.100716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f0117f-add4-4439-a2ed-c158884cc15a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:40.102432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d9f0117f-add4-4439-a2ed-c158884cc15a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:40.159539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:40.216208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f0117f-add4-4439-a2ed-c158884cc15a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:46:40.217814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f0117f-add4-4439-a2ed-c158884cc15a service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:46:40.237219 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 660/1311] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:46:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:46:42.686994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de59f622-fe59-489d-bd2f-0f4a5f3b5eb8 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:42.687503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de59f622-fe59-489d-bd2f-0f4a5f3b5eb8 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:42.687503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de59f622-fe59-489d-bd2f-0f4a5f3b5eb8 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:42.688770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 652/1312] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:46:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:46:42.693850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01eab0c-fa46-4ca7-8d53-dc4e2acebc32 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:42.694265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01eab0c-fa46-4ca7-8d53-dc4e2acebc32 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:42.694549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01eab0c-fa46-4ca7-8d53-dc4e2acebc32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:42.765551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b01eab0c-fa46-4ca7-8d53-dc4e2acebc32 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:46:42.766780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b01eab0c-fa46-4ca7-8d53-dc4e2acebc32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:42.861425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 661/1313] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:46:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:46:43.158280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b80bea-a457-4711-84bd-e23aa01c6121 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:43.158566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b80bea-a457-4711-84bd-e23aa01c6121 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:43.158845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b80bea-a457-4711-84bd-e23aa01c6121 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:43.160160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 653/1314] 10.222.0.25 () {54 vars in 964 bytes} [Sun Oct 2 00:46:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:46:43.166318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbd0a40-78aa-4868-9157-d6b87cc9dcd9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:43.166595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbd0a40-78aa-4868-9157-d6b87cc9dcd9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:43.166861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbd0a40-78aa-4868-9157-d6b87cc9dcd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:43.180079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:43.248718 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fbbd0a40-78aa-4868-9157-d6b87cc9dcd9 None None] MFA Rules not processed for user `e75811ee9ed7490985fed336ab9feb19`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:46:43.250213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fbbd0a40-78aa-4868-9157-d6b87cc9dcd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:43.324486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 662/1315] 10.222.0.25 () {58 vars in 1061 bytes} [Sun Oct 2 00:46:43 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:46:43.333320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b80bea-a457-4711-84bd-e23aa01c6121 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:46:43.335483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a3b80bea-a457-4711-84bd-e23aa01c6121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:43.440311 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b80bea-a457-4711-84bd-e23aa01c6121 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:46:43.511776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:43.513942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c17b02-17d0-4ebd-af11-be98cdfd2bfd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:46:43.514587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c17b02-17d0-4ebd-af11-be98cdfd2bfd service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:43.515230 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c17b02-17d0-4ebd-af11-be98cdfd2bfd service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:43.515891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c17b02-17d0-4ebd-af11-be98cdfd2bfd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:43.517714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-15c17b02-17d0-4ebd-af11-be98cdfd2bfd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:43.623133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c17b02-17d0-4ebd-af11-be98cdfd2bfd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:46:43.624483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c17b02-17d0-4ebd-af11-be98cdfd2bfd service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:46:43.642382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 654/1316] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:46:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:46:46.282622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fef20f3-a20f-4c9f-a414-404da6372432 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:46.283127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fef20f3-a20f-4c9f-a414-404da6372432 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:46.283127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fef20f3-a20f-4c9f-a414-404da6372432 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:46.284416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 663/1317] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:46:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:46:46.290262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf94530-f462-4b67-8282-2991f252793e None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:46.290675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf94530-f462-4b67-8282-2991f252793e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:46.290805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf94530-f462-4b67-8282-2991f252793e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:46.354784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-faf94530-f462-4b67-8282-2991f252793e None None] MFA Rules not processed for user `d7ae9bf858ce45f1887ed9fbf2036581`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:46:46.355984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-faf94530-f462-4b67-8282-2991f252793e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:46.406159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:46.449555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 655/1318] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:46:46 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:46:46.741966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fef20f3-a20f-4c9f-a414-404da6372432 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:46:46.743550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8fef20f3-a20f-4c9f-a414-404da6372432 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:46.844163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fef20f3-a20f-4c9f-a414-404da6372432 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:46:46.846903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:46.916099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56779693-e800-4b62-845a-0ca72e7c01a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:46:46.916690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56779693-e800-4b62-845a-0ca72e7c01a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:46.916958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56779693-e800-4b62-845a-0ca72e7c01a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:46.917222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56779693-e800-4b62-845a-0ca72e7c01a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:46.918825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-56779693-e800-4b62-845a-0ca72e7c01a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:47.023596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56779693-e800-4b62-845a-0ca72e7c01a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:46:47.025031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56779693-e800-4b62-845a-0ca72e7c01a6 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:46:47.043027 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 664/1319] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:46:46 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:46:49.697058 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82321ab2-ea01-4689-8ab2-8f1dc2f7f15d None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:49.697622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82321ab2-ea01-4689-8ab2-8f1dc2f7f15d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:49.697622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82321ab2-ea01-4689-8ab2-8f1dc2f7f15d None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:49.699025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 656/1320] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:46:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:46:49.704114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ea9eaa-c823-4a27-8857-c15ea0bc414d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:49.704523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ea9eaa-c823-4a27-8857-c15ea0bc414d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:49.704663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ea9eaa-c823-4a27-8857-c15ea0bc414d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:49.745202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:49.770485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-25ea9eaa-c823-4a27-8857-c15ea0bc414d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:46:49.772204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-25ea9eaa-c823-4a27-8857-c15ea0bc414d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:49.873885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 665/1321] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:46:49 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:46:50.155380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82321ab2-ea01-4689-8ab2-8f1dc2f7f15d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:46:50.156953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-82321ab2-ea01-4689-8ab2-8f1dc2f7f15d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:50.257130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82321ab2-ea01-4689-8ab2-8f1dc2f7f15d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:46:50.327989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50daf8cd-a5bb-4809-a7be-27049abd42a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:46:50.328513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50daf8cd-a5bb-4809-a7be-27049abd42a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:50.328789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50daf8cd-a5bb-4809-a7be-27049abd42a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:50.329059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50daf8cd-a5bb-4809-a7be-27049abd42a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:50.330642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-50daf8cd-a5bb-4809-a7be-27049abd42a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:50.337323 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:50.436864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50daf8cd-a5bb-4809-a7be-27049abd42a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:46:50.438341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50daf8cd-a5bb-4809-a7be-27049abd42a8 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:46:50.456212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 657/1322] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:46:50 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:46:53.222418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b11d022-8bec-47bc-b405-db7b7be54d7d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:53.222982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b11d022-8bec-47bc-b405-db7b7be54d7d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:53.222982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b11d022-8bec-47bc-b405-db7b7be54d7d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:53.224443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 666/1323] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:46:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:46:53.229906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7259acd8-e32b-4036-bfa6-95be2e962d1c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:53.230326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7259acd8-e32b-4036-bfa6-95be2e962d1c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:53.230609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7259acd8-e32b-4036-bfa6-95be2e962d1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:53.300458 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-7259acd8-e32b-4036-bfa6-95be2e962d1c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:46:53.301599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7259acd8-e32b-4036-bfa6-95be2e962d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:53.363249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:53.407163 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 658/1324] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:46:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:46:53.690867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b11d022-8bec-47bc-b405-db7b7be54d7d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:46:53.692373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5b11d022-8bec-47bc-b405-db7b7be54d7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:53.793982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b11d022-8bec-47bc-b405-db7b7be54d7d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:46:53.807150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:53.870365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8132ec39-f725-4bb3-8cff-d3b1ab414280 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:46:53.870972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132ec39-f725-4bb3-8cff-d3b1ab414280 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:53.871252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132ec39-f725-4bb3-8cff-d3b1ab414280 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:53.871524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8132ec39-f725-4bb3-8cff-d3b1ab414280 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:53.873104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8132ec39-f725-4bb3-8cff-d3b1ab414280 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:53.990181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8132ec39-f725-4bb3-8cff-d3b1ab414280 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:46:53.991587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8132ec39-f725-4bb3-8cff-d3b1ab414280 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:46:54.010085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 667/1325] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:46:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:46:56.229907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bffb7e8-6d99-4928-83a6-bdec245a6a09 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:56.230493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bffb7e8-6d99-4928-83a6-bdec245a6a09 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:56.230493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bffb7e8-6d99-4928-83a6-bdec245a6a09 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:56.231822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 659/1326] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:46:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:46:56.237201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384072f2-ff84-467c-8c3a-f461163f3872 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:56.237623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384072f2-ff84-467c-8c3a-f461163f3872 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:56.237810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384072f2-ff84-467c-8c3a-f461163f3872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:56.285715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:56.305051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-384072f2-ff84-467c-8c3a-f461163f3872 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:46:56.306280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-384072f2-ff84-467c-8c3a-f461163f3872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:56.405512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 668/1327] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:46:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:46:56.691356 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bffb7e8-6d99-4928-83a6-bdec245a6a09 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:46:56.692904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4bffb7e8-6d99-4928-83a6-bdec245a6a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:56.793893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bffb7e8-6d99-4928-83a6-bdec245a6a09 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:46:56.865742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec4e732-8356-4f11-b67c-d542049c48a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:46:56.866367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec4e732-8356-4f11-b67c-d542049c48a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:56.866648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec4e732-8356-4f11-b67c-d542049c48a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:56.866914 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec4e732-8356-4f11-b67c-d542049c48a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:56.868474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2ec4e732-8356-4f11-b67c-d542049c48a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:56.885706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:56.976440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec4e732-8356-4f11-b67c-d542049c48a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:46:56.977779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec4e732-8356-4f11-b67c-d542049c48a7 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:46:56.996238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 660/1328] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:46:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:46:59.187985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478831ff-d925-4f18-a2a2-899a127b82c1 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:59.188489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478831ff-d925-4f18-a2a2-899a127b82c1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:59.188489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478831ff-d925-4f18-a2a2-899a127b82c1 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:59.189769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 669/1329] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:46:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:46:59.195052 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf52dc-8bdd-4fe6-bb49-df0e795e32e4 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:59.195425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf52dc-8bdd-4fe6-bb49-df0e795e32e4 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:59.195591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf52dc-8bdd-4fe6-bb49-df0e795e32e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:59.261294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-66bf52dc-8bdd-4fe6-bb49-df0e795e32e4 None None] MFA Rules not processed for user `d7ae9bf858ce45f1887ed9fbf2036581`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:46:59.262417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-66bf52dc-8bdd-4fe6-bb49-df0e795e32e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:59.327669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:59.359355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 661/1330] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:46:59 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:46:59.643889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478831ff-d925-4f18-a2a2-899a127b82c1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:46:59.645624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-478831ff-d925-4f18-a2a2-899a127b82c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:59.751622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478831ff-d925-4f18-a2a2-899a127b82c1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:46:59.772674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:46:59.826216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2047c89c-b8e1-4ad0-8c51-e8407c0c36ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:46:59.826811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2047c89c-b8e1-4ad0-8c51-e8407c0c36ad service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:46:59.827087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2047c89c-b8e1-4ad0-8c51-e8407c0c36ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:46:59.827354 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2047c89c-b8e1-4ad0-8c51-e8407c0c36ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:46:59.828922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2047c89c-b8e1-4ad0-8c51-e8407c0c36ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:46:59.936499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2047c89c-b8e1-4ad0-8c51-e8407c0c36ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:46:59.937941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2047c89c-b8e1-4ad0-8c51-e8407c0c36ad service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:46:59.956906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 670/1331] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:46:59 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:02.357721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12af7249-8baa-4ace-8808-3575b3b8e482 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:02.358259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12af7249-8baa-4ace-8808-3575b3b8e482 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:02.358259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12af7249-8baa-4ace-8808-3575b3b8e482 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:02.359548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 662/1332] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:02.365289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479d8898-4cc2-4fff-bebb-b9dbbe764798 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:02.365686 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479d8898-4cc2-4fff-bebb-b9dbbe764798 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:02.365825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479d8898-4cc2-4fff-bebb-b9dbbe764798 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:02.427594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:02.430491 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-479d8898-4cc2-4fff-bebb-b9dbbe764798 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:02.432291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-479d8898-4cc2-4fff-bebb-b9dbbe764798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:02.530800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 671/1333] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:02.814817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12af7249-8baa-4ace-8808-3575b3b8e482 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:02.816338 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-12af7249-8baa-4ace-8808-3575b3b8e482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:02.916367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12af7249-8baa-4ace-8808-3575b3b8e482 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:02.992026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a87d3a4-7453-48b6-ae8b-a0e0d7883641 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:02.992623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a87d3a4-7453-48b6-ae8b-a0e0d7883641 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:02.992896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a87d3a4-7453-48b6-ae8b-a0e0d7883641 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:02.993162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a87d3a4-7453-48b6-ae8b-a0e0d7883641 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:02.994746 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2a87d3a4-7453-48b6-ae8b-a0e0d7883641 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:03.020841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:03.101272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a87d3a4-7453-48b6-ae8b-a0e0d7883641 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:03.102719 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a87d3a4-7453-48b6-ae8b-a0e0d7883641 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:03.120623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 663/1334] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:05.308223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2477b448-d0d7-499d-b1b2-f27fde15d3cb None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:05.308740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2477b448-d0d7-499d-b1b2-f27fde15d3cb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:05.308740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2477b448-d0d7-499d-b1b2-f27fde15d3cb None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:05.310008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 672/1335] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:05.315324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f307ebe-113e-4c17-b809-e0fbcbf3066c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:05.315704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f307ebe-113e-4c17-b809-e0fbcbf3066c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:05.315872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f307ebe-113e-4c17-b809-e0fbcbf3066c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:05.379801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9f307ebe-113e-4c17-b809-e0fbcbf3066c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:05.380989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9f307ebe-113e-4c17-b809-e0fbcbf3066c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:05.454364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:05.477313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 664/1336] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:05.760970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2477b448-d0d7-499d-b1b2-f27fde15d3cb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:05.762873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2477b448-d0d7-499d-b1b2-f27fde15d3cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:05.865412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2477b448-d0d7-499d-b1b2-f27fde15d3cb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:05.892816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:05.937277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d753a6a8-5be2-4b32-843a-0edd28fd2674 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:05.937873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d753a6a8-5be2-4b32-843a-0edd28fd2674 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:05.938182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d753a6a8-5be2-4b32-843a-0edd28fd2674 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:05.938454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d753a6a8-5be2-4b32-843a-0edd28fd2674 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:05.940045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d753a6a8-5be2-4b32-843a-0edd28fd2674 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:06.047393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d753a6a8-5be2-4b32-843a-0edd28fd2674 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:06.048831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d753a6a8-5be2-4b32-843a-0edd28fd2674 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:06.066525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 673/1337] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:08.581376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831cdd35-c06b-4e26-874d-fcee051732b0 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:08.581924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831cdd35-c06b-4e26-874d-fcee051732b0 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:08.581924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831cdd35-c06b-4e26-874d-fcee051732b0 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:08.583853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 665/1338] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:08.588903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e487ac7-f511-4c13-8f44-510d4868f761 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:08.589324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e487ac7-f511-4c13-8f44-510d4868f761 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:08.589437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e487ac7-f511-4c13-8f44-510d4868f761 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:08.653222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1e487ac7-f511-4c13-8f44-510d4868f761 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:08.654428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1e487ac7-f511-4c13-8f44-510d4868f761 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:08.662431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:08.748472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 674/1339] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:08 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:09.029423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831cdd35-c06b-4e26-874d-fcee051732b0 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:09.031021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-831cdd35-c06b-4e26-874d-fcee051732b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:09.133770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831cdd35-c06b-4e26-874d-fcee051732b0 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:09.205261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a50e666-daa6-4422-aaaa-c2b3cf7a1992 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:09.205850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a50e666-daa6-4422-aaaa-c2b3cf7a1992 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:09.206172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a50e666-daa6-4422-aaaa-c2b3cf7a1992 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:09.206440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a50e666-daa6-4422-aaaa-c2b3cf7a1992 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:09.208059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1a50e666-daa6-4422-aaaa-c2b3cf7a1992 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:09.241934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:09.315858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a50e666-daa6-4422-aaaa-c2b3cf7a1992 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:09.317238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a50e666-daa6-4422-aaaa-c2b3cf7a1992 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:09.337683 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 666/1340] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:09 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:13.387891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfe182a-d4c7-433c-ae65-7669105784e9 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:13.388411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfe182a-d4c7-433c-ae65-7669105784e9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:13.388411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfe182a-d4c7-433c-ae65-7669105784e9 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:13.389728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 675/1341] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:13.395161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd610a57-eca7-46d9-8b53-89014a446221 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:13.395548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd610a57-eca7-46d9-8b53-89014a446221 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:13.395735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd610a57-eca7-46d9-8b53-89014a446221 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:13.461110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-cd610a57-eca7-46d9-8b53-89014a446221 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:13.462254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cd610a57-eca7-46d9-8b53-89014a446221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:13.546679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:13.557909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 667/1342] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:13.840110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfe182a-d4c7-433c-ae65-7669105784e9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:13.841700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1bfe182a-d4c7-433c-ae65-7669105784e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:13.943476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfe182a-d4c7-433c-ae65-7669105784e9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:13.980428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:14.016344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1664f11a-681b-497e-b91d-ad3fbdb58a1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:14.016943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1664f11a-681b-497e-b91d-ad3fbdb58a1f service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:14.017217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1664f11a-681b-497e-b91d-ad3fbdb58a1f service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:14.017485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1664f11a-681b-497e-b91d-ad3fbdb58a1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:14.019089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1664f11a-681b-497e-b91d-ad3fbdb58a1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:14.130725 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1664f11a-681b-497e-b91d-ad3fbdb58a1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:14.132093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1664f11a-681b-497e-b91d-ad3fbdb58a1f service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:14.150333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 676/1343] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:17.231839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac52563e-2dee-4b2d-bff0-eca7963ced8c None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:17.232389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac52563e-2dee-4b2d-bff0-eca7963ced8c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:17.232389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac52563e-2dee-4b2d-bff0-eca7963ced8c None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:17.233694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 668/1344] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:17.239325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ec7843-2a69-4a87-a26b-1d2de8357f3c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:17.239743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ec7843-2a69-4a87-a26b-1d2de8357f3c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:17.239837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ec7843-2a69-4a87-a26b-1d2de8357f3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:17.306364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f8ec7843-2a69-4a87-a26b-1d2de8357f3c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:17.307475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f8ec7843-2a69-4a87-a26b-1d2de8357f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:17.325895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:17.407487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 677/1345] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:17 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:17.686951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac52563e-2dee-4b2d-bff0-eca7963ced8c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:17.688777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ac52563e-2dee-4b2d-bff0-eca7963ced8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:17.792681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac52563e-2dee-4b2d-bff0-eca7963ced8c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:17.884999 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f86e29-8c75-4e35-8d76-8819891fbdac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:17.885598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f86e29-8c75-4e35-8d76-8819891fbdac service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:17.885871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f86e29-8c75-4e35-8d76-8819891fbdac service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:17.886227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f86e29-8c75-4e35-8d76-8819891fbdac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:17.887785 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-27f86e29-8c75-4e35-8d76-8819891fbdac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:17.934844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:18.003580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f86e29-8c75-4e35-8d76-8819891fbdac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:18.005615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f86e29-8c75-4e35-8d76-8819891fbdac service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:18.023990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 669/1346] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:17 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:22.306503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79232374-d569-44f7-903d-a3ced3204417 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:22.307016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79232374-d569-44f7-903d-a3ced3204417 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:22.307096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79232374-d569-44f7-903d-a3ced3204417 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:22.308470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 678/1347] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:22.314187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f3b357-d6e5-4b1b-9844-afbaab0588cb None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:22.314612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f3b357-d6e5-4b1b-9844-afbaab0588cb None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:22.314740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f3b357-d6e5-4b1b-9844-afbaab0588cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:22.382765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-08f3b357-d6e5-4b1b-9844-afbaab0588cb None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:22.383970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-08f3b357-d6e5-4b1b-9844-afbaab0588cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:22.480613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:22.486104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 670/1348] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:22.777835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79232374-d569-44f7-903d-a3ced3204417 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:22.779407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-79232374-d569-44f7-903d-a3ced3204417 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:22.880087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79232374-d569-44f7-903d-a3ced3204417 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:22.926428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:22.951424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673e7b59-ff4d-47cd-9919-79ebb9bd82c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:22.952008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673e7b59-ff4d-47cd-9919-79ebb9bd82c5 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:22.952278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673e7b59-ff4d-47cd-9919-79ebb9bd82c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:22.952543 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673e7b59-ff4d-47cd-9919-79ebb9bd82c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:22.954182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-673e7b59-ff4d-47cd-9919-79ebb9bd82c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:23.061615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673e7b59-ff4d-47cd-9919-79ebb9bd82c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:23.063070 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673e7b59-ff4d-47cd-9919-79ebb9bd82c5 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:23.081162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 679/1349] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:27.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ab795a-4bf4-4759-b9ba-d8e2e5ecbd2b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:27.910752 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ab795a-4bf4-4759-b9ba-d8e2e5ecbd2b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:27.910847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ab795a-4bf4-4759-b9ba-d8e2e5ecbd2b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:27.912189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 671/1350] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:47:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:27.917895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd064145-4653-4c4f-83a8-f691e5d4a6f1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:27.918343 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd064145-4653-4c4f-83a8-f691e5d4a6f1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:27.918463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd064145-4653-4c4f-83a8-f691e5d4a6f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:28.015855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cd064145-4653-4c4f-83a8-f691e5d4a6f1 None None] MFA Rules not processed for user `5eeff4f989ae4338a182bfdadd3a8b63`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:28.018024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cd064145-4653-4c4f-83a8-f691e5d4a6f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:28.061480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:28.195952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 680/1351] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:47:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:32.284544 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b973d11a-7f18-42f5-b94c-d3c8d34be1df None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:32.285153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b973d11a-7f18-42f5-b94c-d3c8d34be1df None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:32.285367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b973d11a-7f18-42f5-b94c-d3c8d34be1df None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:32.287594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 672/1352] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:47:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:32.292760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eccd4ff-49df-4248-9a8d-91d94393d7aa None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:32.293167 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eccd4ff-49df-4248-9a8d-91d94393d7aa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:32.293330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eccd4ff-49df-4248-9a8d-91d94393d7aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:32.376136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3eccd4ff-49df-4248-9a8d-91d94393d7aa None None] MFA Rules not processed for user `5eeff4f989ae4338a182bfdadd3a8b63`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:32.377342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3eccd4ff-49df-4248-9a8d-91d94393d7aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:32.453571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:32.494455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 681/1353] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:47:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:33.885059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aeb19b-cc6e-41d0-8612-3e45c138f2ab None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:33.885588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aeb19b-cc6e-41d0-8612-3e45c138f2ab None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:33.885588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aeb19b-cc6e-41d0-8612-3e45c138f2ab None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:33.886955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 673/1354] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:47:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:33.892164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0702c880-7436-45f6-88ec-8224055e75ab None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:33.892612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0702c880-7436-45f6-88ec-8224055e75ab None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:33.892711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0702c880-7436-45f6-88ec-8224055e75ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:33.967719 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0702c880-7436-45f6-88ec-8224055e75ab None None] MFA Rules not processed for user `5eeff4f989ae4338a182bfdadd3a8b63`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:33.968908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0702c880-7436-45f6-88ec-8224055e75ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:34.079507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 682/1355] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:47:33 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:47:37.596338 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03e54d3-9a7f-4429-8d3e-0cb68ff27e39 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:37.596903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03e54d3-9a7f-4429-8d3e-0cb68ff27e39 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:37.596903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03e54d3-9a7f-4429-8d3e-0cb68ff27e39 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:37.598486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 674/1356] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:37.603735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a59f95a-2d37-4065-b128-9eb53a37d6e5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:37.604142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a59f95a-2d37-4065-b128-9eb53a37d6e5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:37.604268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a59f95a-2d37-4065-b128-9eb53a37d6e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:37.641926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:37.677084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9a59f95a-2d37-4065-b128-9eb53a37d6e5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:37.678224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9a59f95a-2d37-4065-b128-9eb53a37d6e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:37.779039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 683/1357] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:38.079416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba9c783-008f-4259-890c-22876fc75e12 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:38.079866 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba9c783-008f-4259-890c-22876fc75e12 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:38.080309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba9c783-008f-4259-890c-22876fc75e12 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:38.082095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 675/1358] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 2 00:47:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:38.091283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0845e5-9a2a-4c70-b9c7-bd62be52d38c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:38.091889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0845e5-9a2a-4c70-b9c7-bd62be52d38c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:38.092372 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0845e5-9a2a-4c70-b9c7-bd62be52d38c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:38.183194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ea0845e5-9a2a-4c70-b9c7-bd62be52d38c None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:38.184484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ea0845e5-9a2a-4c70-b9c7-bd62be52d38c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:38.233905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:38.306176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 684/1359] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 2 00:47:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:38.313693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba9c783-008f-4259-890c-22876fc75e12 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:38.315671 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-aba9c783-008f-4259-890c-22876fc75e12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:38.474958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba9c783-008f-4259-890c-22876fc75e12 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:38.550910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:38.592413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47949b1-60dc-4018-9720-80aca600a300 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:38.593049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47949b1-60dc-4018-9720-80aca600a300 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:38.593327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47949b1-60dc-4018-9720-80aca600a300 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:38.593593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47949b1-60dc-4018-9720-80aca600a300 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:38.595185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a47949b1-60dc-4018-9720-80aca600a300 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:38.691445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47949b1-60dc-4018-9720-80aca600a300 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:38.693294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47949b1-60dc-4018-9720-80aca600a300 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:38.713360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 676/1360] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:47:41.090706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a66beb-9d5f-4a73-84cb-579183f83a3e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:41.091281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a66beb-9d5f-4a73-84cb-579183f83a3e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:41.091281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a66beb-9d5f-4a73-84cb-579183f83a3e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:41.092567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 685/1361] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:41.098787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2729b015-cba1-49c4-abca-cc67440c4b76 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:41.099615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2729b015-cba1-49c4-abca-cc67440c4b76 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:41.100071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2729b015-cba1-49c4-abca-cc67440c4b76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:41.176175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-2729b015-cba1-49c4-abca-cc67440c4b76 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:41.178160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2729b015-cba1-49c4-abca-cc67440c4b76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:41.230684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:41.302109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 677/1362] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:41.601571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a66beb-9d5f-4a73-84cb-579183f83a3e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:41.603130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e0a66beb-9d5f-4a73-84cb-579183f83a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:41.781994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a66beb-9d5f-4a73-84cb-579183f83a3e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:41.794432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:41.933353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4976c74-1633-476b-b444-0a981be7c87d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:41.934111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4976c74-1633-476b-b444-0a981be7c87d service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:41.934351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4976c74-1633-476b-b444-0a981be7c87d service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:41.934691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4976c74-1633-476b-b444-0a981be7c87d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:41.936227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d4976c74-1633-476b-b444-0a981be7c87d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:42.028311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4976c74-1633-476b-b444-0a981be7c87d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:42.031824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4976c74-1633-476b-b444-0a981be7c87d service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:42.052354 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 686/1363] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:44.571046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea598ed-7eb1-4e75-a795-8e94ca23d24c None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:44.571555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea598ed-7eb1-4e75-a795-8e94ca23d24c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:44.571555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea598ed-7eb1-4e75-a795-8e94ca23d24c None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:44.573039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 678/1364] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:44.578321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f9daa-6f0f-4b6f-8bc7-956640273bcb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:44.578754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f9daa-6f0f-4b6f-8bc7-956640273bcb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:44.578925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7f9daa-6f0f-4b6f-8bc7-956640273bcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:44.594430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:44.649262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1e7f9daa-6f0f-4b6f-8bc7-956640273bcb None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:44.650497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1e7f9daa-6f0f-4b6f-8bc7-956640273bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:44.752512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 687/1365] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:45.047350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cab479-dab7-40a8-b6be-b5cd2b54963b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:45.047693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cab479-dab7-40a8-b6be-b5cd2b54963b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:45.047896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cab479-dab7-40a8-b6be-b5cd2b54963b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:45.049999 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 679/1366] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 2 00:47:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:45.056174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc704200-6fbe-45e3-b3d8-95a48a7ef6f6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:45.056558 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc704200-6fbe-45e3-b3d8-95a48a7ef6f6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:45.056931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc704200-6fbe-45e3-b3d8-95a48a7ef6f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:45.137687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-bc704200-6fbe-45e3-b3d8-95a48a7ef6f6 None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:45.139080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bc704200-6fbe-45e3-b3d8-95a48a7ef6f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:45.178331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:45.262923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 688/1367] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 2 00:47:45 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:45.272335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cab479-dab7-40a8-b6be-b5cd2b54963b None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:45.274160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-25cab479-dab7-40a8-b6be-b5cd2b54963b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:45.431896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cab479-dab7-40a8-b6be-b5cd2b54963b None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:45.453870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:45.553809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d44188-7391-4b85-bccf-9efd74db73dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:45.554451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d44188-7391-4b85-bccf-9efd74db73dc service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:45.554729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d44188-7391-4b85-bccf-9efd74db73dc service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:45.554996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d44188-7391-4b85-bccf-9efd74db73dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:45.556701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-74d44188-7391-4b85-bccf-9efd74db73dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:45.656726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d44188-7391-4b85-bccf-9efd74db73dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:45.658904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d44188-7391-4b85-bccf-9efd74db73dc service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:45.678103 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 680/1368] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:45.812858 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ec99f8-454c-422f-880d-8920063ca67d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:45.813878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ec99f8-454c-422f-880d-8920063ca67d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:45.814201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ec99f8-454c-422f-880d-8920063ca67d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:45.816293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 689/1369] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:45 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:45.822655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ba264-04d2-4f68-8016-f7c766899a73 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:45.822938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ba264-04d2-4f68-8016-f7c766899a73 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:45.823204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5ba264-04d2-4f68-8016-f7c766899a73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:45.860787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:45.909737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-cf5ba264-04d2-4f68-8016-f7c766899a73 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:45.911001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cf5ba264-04d2-4f68-8016-f7c766899a73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:46.026600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 681/1370] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:46.353730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064792ea-30a4-4aab-8f36-1cf974b3f813 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:46.353730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064792ea-30a4-4aab-8f36-1cf974b3f813 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:46.353730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064792ea-30a4-4aab-8f36-1cf974b3f813 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:46.354802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 690/1371] 10.222.0.25 () {54 vars in 963 bytes} [Sun Oct 2 00:47:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:46.359576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6ebd93-b15d-4b9b-b678-00922aa2da2c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:46.359863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6ebd93-b15d-4b9b-b678-00922aa2da2c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:46.360133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6ebd93-b15d-4b9b-b678-00922aa2da2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:46.438039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-7f6ebd93-b15d-4b9b-b678-00922aa2da2c None None] MFA Rules not processed for user `eb325d3c277b45c2a5c4e0e94078be29`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:46.439256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7f6ebd93-b15d-4b9b-b678-00922aa2da2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:46.489518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:46.530223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 682/1372] 10.222.0.25 () {58 vars in 1060 bytes} [Sun Oct 2 00:47:46 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:46.532340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064792ea-30a4-4aab-8f36-1cf974b3f813 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:46.534208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-064792ea-30a4-4aab-8f36-1cf974b3f813 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:46.653122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064792ea-30a4-4aab-8f36-1cf974b3f813 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:46.699011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:46.737362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da06e7f5-3c41-445c-a7b5-521e72906a70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:46.738141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da06e7f5-3c41-445c-a7b5-521e72906a70 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:46.738518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da06e7f5-3c41-445c-a7b5-521e72906a70 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:46.738886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da06e7f5-3c41-445c-a7b5-521e72906a70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:46.742206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-da06e7f5-3c41-445c-a7b5-521e72906a70 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:46.882165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da06e7f5-3c41-445c-a7b5-521e72906a70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:46.883601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da06e7f5-3c41-445c-a7b5-521e72906a70 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:46.904448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 691/1373] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:47.919014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5497dd7-7cad-403b-9214-dc6af5262981 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:47.919532 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5497dd7-7cad-403b-9214-dc6af5262981 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:47.919532 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5497dd7-7cad-403b-9214-dc6af5262981 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:47.920896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 683/1374] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:47.926754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e41a2a-cf0b-4365-ac97-c91bec27f926 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:47.927214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e41a2a-cf0b-4365-ac97-c91bec27f926 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:47.927316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e41a2a-cf0b-4365-ac97-c91bec27f926 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:47.991141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-38e41a2a-cf0b-4365-ac97-c91bec27f926 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:47.992329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-38e41a2a-cf0b-4365-ac97-c91bec27f926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:48.012254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:48.094165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 692/1375] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:48.389004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5497dd7-7cad-403b-9214-dc6af5262981 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:48.390565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f5497dd7-7cad-403b-9214-dc6af5262981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:48.569331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5497dd7-7cad-403b-9214-dc6af5262981 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:48.621957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:48.686618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc3d483-401a-4caf-996e-279c52fa0c81 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:48.687248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc3d483-401a-4caf-996e-279c52fa0c81 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:48.687595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc3d483-401a-4caf-996e-279c52fa0c81 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:48.687932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc3d483-401a-4caf-996e-279c52fa0c81 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:48.690580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4fc3d483-401a-4caf-996e-279c52fa0c81 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:48.781816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc3d483-401a-4caf-996e-279c52fa0c81 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:48.783715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc3d483-401a-4caf-996e-279c52fa0c81 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:48.811131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 684/1376] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:51.010607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e312aa3-c74f-4d95-ac41-645cf340012e None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:51.011159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e312aa3-c74f-4d95-ac41-645cf340012e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:51.011159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e312aa3-c74f-4d95-ac41-645cf340012e None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:51.012614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 693/1377] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:51.017864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dfa300-f09c-475a-997a-fa5f47518f1c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:51.018331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dfa300-f09c-475a-997a-fa5f47518f1c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:51.018509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dfa300-f09c-475a-997a-fa5f47518f1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:51.095302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-32dfa300-f09c-475a-997a-fa5f47518f1c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:51.096543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-32dfa300-f09c-475a-997a-fa5f47518f1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:51.103724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:51.198380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 685/1378] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:51.487132 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e312aa3-c74f-4d95-ac41-645cf340012e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:51.488749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7e312aa3-c74f-4d95-ac41-645cf340012e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:51.643988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e312aa3-c74f-4d95-ac41-645cf340012e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:51.647980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:51.766549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251cf379-bc46-432d-b5ea-d18c8db18e26 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:51.767157 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251cf379-bc46-432d-b5ea-d18c8db18e26 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:51.767460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251cf379-bc46-432d-b5ea-d18c8db18e26 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:51.767736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251cf379-bc46-432d-b5ea-d18c8db18e26 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:51.769299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-251cf379-bc46-432d-b5ea-d18c8db18e26 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:51.864269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251cf379-bc46-432d-b5ea-d18c8db18e26 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:51.866039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251cf379-bc46-432d-b5ea-d18c8db18e26 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:51.885486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 694/1379] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:54.086121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad255e34-23f4-4ea1-a038-8e696ca8a63f None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:54.086674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad255e34-23f4-4ea1-a038-8e696ca8a63f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:54.086674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad255e34-23f4-4ea1-a038-8e696ca8a63f None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:54.090105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 686/1380] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:54.093392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e259300-a61f-4920-8b4d-8c7c683be77a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:54.093814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e259300-a61f-4920-8b4d-8c7c683be77a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:54.093910 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e259300-a61f-4920-8b4d-8c7c683be77a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:54.106095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:54.161296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7e259300-a61f-4920-8b4d-8c7c683be77a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:54.162522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7e259300-a61f-4920-8b4d-8c7c683be77a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:54.270095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 695/1381] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:54.561339 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad255e34-23f4-4ea1-a038-8e696ca8a63f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:54.563259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ad255e34-23f4-4ea1-a038-8e696ca8a63f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:54.722286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:54.732636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad255e34-23f4-4ea1-a038-8e696ca8a63f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:54.848119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99fafa60-56c2-43ed-874e-96d887f0f369 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:54.848600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fafa60-56c2-43ed-874e-96d887f0f369 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:54.848873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fafa60-56c2-43ed-874e-96d887f0f369 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:54.849150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99fafa60-56c2-43ed-874e-96d887f0f369 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:54.850803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-99fafa60-56c2-43ed-874e-96d887f0f369 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:54.943814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fafa60-56c2-43ed-874e-96d887f0f369 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:54.945469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99fafa60-56c2-43ed-874e-96d887f0f369 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:54.956366 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:54.964910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 687/1382] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:47:57.504663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17db749e-8e0d-4a06-ad45-421dda3c0cfa None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:57.505344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17db749e-8e0d-4a06-ad45-421dda3c0cfa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:57.505344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17db749e-8e0d-4a06-ad45-421dda3c0cfa None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:57.506631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 696/1383] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:47:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:47:57.512207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174bb703-0828-4ab6-bdfb-2357f03588bd None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:57.512650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174bb703-0828-4ab6-bdfb-2357f03588bd None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:57.512909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174bb703-0828-4ab6-bdfb-2357f03588bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:57.584716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-174bb703-0828-4ab6-bdfb-2357f03588bd None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:47:57.586013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-174bb703-0828-4ab6-bdfb-2357f03588bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:57.690890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 688/1384] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:47:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:47:57.985428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17db749e-8e0d-4a06-ad45-421dda3c0cfa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:47:57.987626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-17db749e-8e0d-4a06-ad45-421dda3c0cfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:58.084087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:58.148376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17db749e-8e0d-4a06-ad45-421dda3c0cfa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:47:58.266733 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a7c899-9aca-4a77-95ec-31d7df4e2d61 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:47:58.267442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a7c899-9aca-4a77-95ec-31d7df4e2d61 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:47:58.267727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a7c899-9aca-4a77-95ec-31d7df4e2d61 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:47:58.268007 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a7c899-9aca-4a77-95ec-31d7df4e2d61 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:47:58.269644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-54a7c899-9aca-4a77-95ec-31d7df4e2d61 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:47:58.323316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:47:58.364477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a7c899-9aca-4a77-95ec-31d7df4e2d61 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:47:58.366191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a7c899-9aca-4a77-95ec-31d7df4e2d61 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:47:58.384770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 697/1385] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:47:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:00.550528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd90f37b-f5a4-48a7-a41e-16620684c0c8 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:00.551075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd90f37b-f5a4-48a7-a41e-16620684c0c8 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:00.551075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd90f37b-f5a4-48a7-a41e-16620684c0c8 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:00.552439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 689/1386] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:48:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:00.557687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e71aa3f-75bd-446d-ad89-f3498ffe550f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:00.558111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e71aa3f-75bd-446d-ad89-f3498ffe550f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:00.558386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e71aa3f-75bd-446d-ad89-f3498ffe550f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:00.623985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7e71aa3f-75bd-446d-ad89-f3498ffe550f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:00.625187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7e71aa3f-75bd-446d-ad89-f3498ffe550f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:00.726276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 698/1387] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:48:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:01.015961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd90f37b-f5a4-48a7-a41e-16620684c0c8 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:01.017495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cd90f37b-f5a4-48a7-a41e-16620684c0c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:01.100171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:01.168769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd90f37b-f5a4-48a7-a41e-16620684c0c8 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:01.284147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9747cf15-a59d-4718-93ac-1d7d994724cc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:01.284666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9747cf15-a59d-4718-93ac-1d7d994724cc service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:01.284937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9747cf15-a59d-4718-93ac-1d7d994724cc service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:01.285202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9747cf15-a59d-4718-93ac-1d7d994724cc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:01.286814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9747cf15-a59d-4718-93ac-1d7d994724cc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:01.325086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:01.377226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9747cf15-a59d-4718-93ac-1d7d994724cc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:01.378874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9747cf15-a59d-4718-93ac-1d7d994724cc service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:01.396948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 690/1388] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:02.175040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79493a1-7fd9-40ca-aa4b-e4efb3ec6cb3 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:02.175567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79493a1-7fd9-40ca-aa4b-e4efb3ec6cb3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:02.175567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79493a1-7fd9-40ca-aa4b-e4efb3ec6cb3 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:02.176914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 699/1389] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:48:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:02.182085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a060154-8b6a-4543-8fcf-b8ce388dfc93 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:02.182470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a060154-8b6a-4543-8fcf-b8ce388dfc93 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:02.182643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a060154-8b6a-4543-8fcf-b8ce388dfc93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:02.248996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4a060154-8b6a-4543-8fcf-b8ce388dfc93 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:02.250230 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4a060154-8b6a-4543-8fcf-b8ce388dfc93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:02.348340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 691/1390] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:48:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:02.647814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702fb40b-fd67-4d44-b748-dd72b0824764 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:02.648280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702fb40b-fd67-4d44-b748-dd72b0824764 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:02.648617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702fb40b-fd67-4d44-b748-dd72b0824764 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:02.650045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 700/1391] 10.222.0.25 () {54 vars in 962 bytes} [Sun Oct 2 00:48:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:02.655435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3af26a-9a71-4fad-914b-52f539620fa9 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:02.655716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3af26a-9a71-4fad-914b-52f539620fa9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:02.655983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3af26a-9a71-4fad-914b-52f539620fa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:02.671493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:02.734281 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-da3af26a-9a71-4fad-914b-52f539620fa9 None None] MFA Rules not processed for user `30fc59f80f544cef8b3869e8d37bb341`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:02.735487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-da3af26a-9a71-4fad-914b-52f539620fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:02.812219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 692/1392] 10.222.0.25 () {58 vars in 1059 bytes} [Sun Oct 2 00:48:02 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:02.821291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702fb40b-fd67-4d44-b748-dd72b0824764 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:02.822833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-702fb40b-fd67-4d44-b748-dd72b0824764 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:02.853598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:02.926576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702fb40b-fd67-4d44-b748-dd72b0824764 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:03.000248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398df392-5754-40df-a198-146d4ffc940a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc59f80f544cef8b3869e8d37bb341', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:03.000853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398df392-5754-40df-a198-146d4ffc940a service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:03.001126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398df392-5754-40df-a198-146d4ffc940a service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:03.001393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398df392-5754-40df-a198-146d4ffc940a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:03.003011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-398df392-5754-40df-a198-146d4ffc940a service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:03.080634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:03.112652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398df392-5754-40df-a198-146d4ffc940a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:03.114149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398df392-5754-40df-a198-146d4ffc940a service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:03.132746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 701/1393] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:03.617890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279038f7-cea2-40d3-93ca-ed3791ff9b0d None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:03.618453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279038f7-cea2-40d3-93ca-ed3791ff9b0d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:03.618453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279038f7-cea2-40d3-93ca-ed3791ff9b0d None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:03.619788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 693/1394] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:48:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:03.625421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb3c753-bcd5-4f9a-859b-fc45e1077498 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:03.625863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb3c753-bcd5-4f9a-859b-fc45e1077498 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:03.625958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb3c753-bcd5-4f9a-859b-fc45e1077498 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:03.692899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3eb3c753-bcd5-4f9a-859b-fc45e1077498 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:03.694165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3eb3c753-bcd5-4f9a-859b-fc45e1077498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:03.812429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 702/1395] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:48:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:03.897743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce30e86-a449-446d-8560-77f5107648af None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:03.898037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce30e86-a449-446d-8560-77f5107648af None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:03.898353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce30e86-a449-446d-8560-77f5107648af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:03.970635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-2ce30e86-a449-446d-8560-77f5107648af None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:03.971877 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2ce30e86-a449-446d-8560-77f5107648af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:04.041992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:04.078107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 694/1396] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 00:48:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:04.099902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb3c753-bcd5-4f9a-859b-fc45e1077498 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:04.101015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3eb3c753-bcd5-4f9a-859b-fc45e1077498 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:04.125668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d4a339-84ca-4444-b97c-894af0472976 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:04.125950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d4a339-84ca-4444-b97c-894af0472976 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:04.126778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d4a339-84ca-4444-b97c-894af0472976 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:04.128121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 695/1397] 10.222.0.25 () {54 vars in 943 bytes} [Sun Oct 2 00:48:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:48:04.134354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1b6a4a-7f6d-4cc1-a771-e1ea2dc635c3 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:04.134634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1b6a4a-7f6d-4cc1-a771-e1ea2dc635c3 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:04.134908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1b6a4a-7f6d-4cc1-a771-e1ea2dc635c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:04.143359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:04.214367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ba1b6a4a-7f6d-4cc1-a771-e1ea2dc635c3 None None] MFA Rules not processed for user `db6c9dbbcf244a6f8bac59d2c9c973bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:04.215574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ba1b6a4a-7f6d-4cc1-a771-e1ea2dc635c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:04.218279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb3c753-bcd5-4f9a-859b-fc45e1077498 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:04.325130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 696/1398] 10.222.0.25 () {58 vars in 1040 bytes} [Sun Oct 2 00:48:04 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:48:04.331943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbe39c2-6624-414e-9391-cce5b9c17dda service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:04.332577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbe39c2-6624-414e-9391-cce5b9c17dda service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:04.332852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbe39c2-6624-414e-9391-cce5b9c17dda service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:04.333122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbe39c2-6624-414e-9391-cce5b9c17dda service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:04.334777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7bbe39c2-6624-414e-9391-cce5b9c17dda service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:04.335476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1b6a4a-7f6d-4cc1-a771-e1ea2dc635c3 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:04.345595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1b6a4a-7f6d-4cc1-a771-e1ea2dc635c3 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:04.370691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:04.371304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:04.429720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbe39c2-6624-414e-9391-cce5b9c17dda service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:04.431347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bbe39c2-6624-414e-9391-cce5b9c17dda service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:04.449905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 703/1399] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:04.453311 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd28676-19fe-4bea-a2cf-4c449199a740 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:04.453922 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd28676-19fe-4bea-a2cf-4c449199a740 service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:04.454236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd28676-19fe-4bea-a2cf-4c449199a740 service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:04.454543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd28676-19fe-4bea-a2cf-4c449199a740 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:04.456164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7cd28676-19fe-4bea-a2cf-4c449199a740 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:04.547641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd28676-19fe-4bea-a2cf-4c449199a740 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:04.549882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd28676-19fe-4bea-a2cf-4c449199a740 service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:04.559127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 697/1400] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 00:48:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 02 00:48:06.653776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01aa2d6-8b92-4ac1-8cab-d2a236b5425f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:06.654468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01aa2d6-8b92-4ac1-8cab-d2a236b5425f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:06.654468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01aa2d6-8b92-4ac1-8cab-d2a236b5425f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:06.655683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 704/1401] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:06.661418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b9ec3e-331a-4718-8830-b7d2dcdb77de None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:06.661840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b9ec3e-331a-4718-8830-b7d2dcdb77de None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:06.661978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b9ec3e-331a-4718-8830-b7d2dcdb77de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:06.758238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-54b9ec3e-331a-4718-8830-b7d2dcdb77de None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:06.759614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-54b9ec3e-331a-4718-8830-b7d2dcdb77de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:06.766486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:06.868076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 698/1402] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:48:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:07.156531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01aa2d6-8b92-4ac1-8cab-d2a236b5425f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:07.158590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b01aa2d6-8b92-4ac1-8cab-d2a236b5425f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:07.439675 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01aa2d6-8b92-4ac1-8cab-d2a236b5425f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:07.460172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:07.574473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b3a37b-c96a-42e9-8a20-58eee589b9d2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:07.575073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b3a37b-c96a-42e9-8a20-58eee589b9d2 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:07.575345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b3a37b-c96a-42e9-8a20-58eee589b9d2 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:07.575612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b3a37b-c96a-42e9-8a20-58eee589b9d2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:07.577205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-21b3a37b-c96a-42e9-8a20-58eee589b9d2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:07.678728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b3a37b-c96a-42e9-8a20-58eee589b9d2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:07.681056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b3a37b-c96a-42e9-8a20-58eee589b9d2 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:07.699811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 705/1403] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 544 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:09.956609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a15967-eca4-4442-910e-9c92e2cf8406 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:09.956609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a15967-eca4-4442-910e-9c92e2cf8406 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:09.956609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a15967-eca4-4442-910e-9c92e2cf8406 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:09.957393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 699/1404] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:48:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:09.962334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4f2a8d-92e3-49d4-bd99-658be6186e52 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:09.962765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4f2a8d-92e3-49d4-bd99-658be6186e52 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:09.962889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4f2a8d-92e3-49d4-bd99-658be6186e52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:09.983322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:10.045840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8a4f2a8d-92e3-49d4-bd99-658be6186e52 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:10.047109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8a4f2a8d-92e3-49d4-bd99-658be6186e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:10.151513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 706/1405] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:48:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:10.444593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a15967-eca4-4442-910e-9c92e2cf8406 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:10.446637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-34a15967-eca4-4442-910e-9c92e2cf8406 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:10.605933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:10.618452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a15967-eca4-4442-910e-9c92e2cf8406 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:10.759100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36591a61-9c96-475a-a12e-b803b9336962 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:10.759711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36591a61-9c96-475a-a12e-b803b9336962 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:10.759987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36591a61-9c96-475a-a12e-b803b9336962 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:10.760254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36591a61-9c96-475a-a12e-b803b9336962 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:10.761864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-36591a61-9c96-475a-a12e-b803b9336962 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:10.870708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36591a61-9c96-475a-a12e-b803b9336962 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:10.872362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36591a61-9c96-475a-a12e-b803b9336962 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:10.883237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:10.893608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 700/1406] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:13.129801 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596ede42-4c07-4abd-b4bc-427269e60555 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:13.130396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596ede42-4c07-4abd-b4bc-427269e60555 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:13.130396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596ede42-4c07-4abd-b4bc-427269e60555 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:13.131700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 707/1407] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:48:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:13.136971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c1fc90-ef51-41c6-b84f-3915cdb50285 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:13.137412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c1fc90-ef51-41c6-b84f-3915cdb50285 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:13.137630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c1fc90-ef51-41c6-b84f-3915cdb50285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:13.224835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-07c1fc90-ef51-41c6-b84f-3915cdb50285 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:13.224835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-07c1fc90-ef51-41c6-b84f-3915cdb50285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:13.361551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 701/1408] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:48:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:13.655889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596ede42-4c07-4abd-b4bc-427269e60555 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:13.657467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-596ede42-4c07-4abd-b4bc-427269e60555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:13.853769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:13.948115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596ede42-4c07-4abd-b4bc-427269e60555 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:14.077307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623baeb9-25c2-4ba8-a6fd-a246a274fece service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:14.077901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623baeb9-25c2-4ba8-a6fd-a246a274fece service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:14.078211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623baeb9-25c2-4ba8-a6fd-a246a274fece service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:14.078489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623baeb9-25c2-4ba8-a6fd-a246a274fece service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:14.080711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-623baeb9-25c2-4ba8-a6fd-a246a274fece service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:14.148865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:14.180939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623baeb9-25c2-4ba8-a6fd-a246a274fece service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:14.182564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623baeb9-25c2-4ba8-a6fd-a246a274fece service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:14.203035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 708/1409] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 548 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:23.358969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=67 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:24.200770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=20 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:26.861766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ad5fc6-188d-45f0-a3cf-a3a9e7e12e78 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:26.862336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ad5fc6-188d-45f0-a3cf-a3a9e7e12e78 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:26.862336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ad5fc6-188d-45f0-a3cf-a3a9e7e12e78 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:26.863584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 702/1410] 10.222.0.25 () {54 vars in 918 bytes} [Sun Oct 2 00:48:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:26.868582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40625d6-211b-4992-9294-c0d9b64aa724 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:26.869026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40625d6-211b-4992-9294-c0d9b64aa724 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:26.869125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40625d6-211b-4992-9294-c0d9b64aa724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:26.952486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d40625d6-211b-4992-9294-c0d9b64aa724 None None] MFA Rules not processed for user `5eeff4f989ae4338a182bfdadd3a8b63`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:26.953673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d40625d6-211b-4992-9294-c0d9b64aa724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:27.056636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 709/1411] 10.222.0.25 () {58 vars in 1015 bytes} [Sun Oct 2 00:48:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:27.070947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b98d2e-9d40-466e-9f82-1b2e0ec6a3e2 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:27.071225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b98d2e-9d40-466e-9f82-1b2e0ec6a3e2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:27.071491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b98d2e-9d40-466e-9f82-1b2e0ec6a3e2 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:27.072808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 703/1412] 10.222.0.25 () {54 vars in 918 bytes} [Sun Oct 2 00:48:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:27.077363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a01f19e-02f0-46cb-b8b0-8c92fe07f58d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:27.077638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a01f19e-02f0-46cb-b8b0-8c92fe07f58d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:27.077905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a01f19e-02f0-46cb-b8b0-8c92fe07f58d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:27.154998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:27.158480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4a01f19e-02f0-46cb-b8b0-8c92fe07f58d None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:27.160181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4a01f19e-02f0-46cb-b8b0-8c92fe07f58d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:27.272752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 710/1413] 10.222.0.25 () {58 vars in 1015 bytes} [Sun Oct 2 00:48:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:29.851381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb688d9-7ca8-48da-999f-0efdb0595db1 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:29.851978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb688d9-7ca8-48da-999f-0efdb0595db1 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:29.851978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb688d9-7ca8-48da-999f-0efdb0595db1 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:29.853261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 704/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:48:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:29.858454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df36fa79-b829-4b5d-b07d-1aa8c5b8227a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:29.858844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df36fa79-b829-4b5d-b07d-1aa8c5b8227a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:29.858998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df36fa79-b829-4b5d-b07d-1aa8c5b8227a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:29.924868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-df36fa79-b829-4b5d-b07d-1aa8c5b8227a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:29.926092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-df36fa79-b829-4b5d-b07d-1aa8c5b8227a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:29.980428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:30.022051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 711/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:48:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:30.317379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb688d9-7ca8-48da-999f-0efdb0595db1 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:30.318985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7fb688d9-7ca8-48da-999f-0efdb0595db1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:30.465677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb688d9-7ca8-48da-999f-0efdb0595db1 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:30.537181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:30.576132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3f3089-cfa0-49a9-a0e5-3881d6885d13 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:30.576761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3f3089-cfa0-49a9-a0e5-3881d6885d13 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:30.577037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3f3089-cfa0-49a9-a0e5-3881d6885d13 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:30.577307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3f3089-cfa0-49a9-a0e5-3881d6885d13 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:30.579499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7f3f3089-cfa0-49a9-a0e5-3881d6885d13 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:30.665857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3f3089-cfa0-49a9-a0e5-3881d6885d13 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:30.667477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3f3089-cfa0-49a9-a0e5-3881d6885d13 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:30.687381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 705/1416] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:31.825769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011e474e-07c7-4cb7-827e-4bc60f991933 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:31.826338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011e474e-07c7-4cb7-827e-4bc60f991933 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:31.826338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011e474e-07c7-4cb7-827e-4bc60f991933 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:31.827665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 712/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:48:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:31.834243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf28fb82-b682-46ae-8af1-b6afe94d841c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:31.834657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf28fb82-b682-46ae-8af1-b6afe94d841c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:31.834810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf28fb82-b682-46ae-8af1-b6afe94d841c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:31.909265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-cf28fb82-b682-46ae-8af1-b6afe94d841c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:31.910496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cf28fb82-b682-46ae-8af1-b6afe94d841c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:31.936880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:32.006886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 706/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:48:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:32.154714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011e474e-07c7-4cb7-827e-4bc60f991933 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:32.156149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-011e474e-07c7-4cb7-827e-4bc60f991933 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:32.297387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011e474e-07c7-4cb7-827e-4bc60f991933 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:32.336432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:32.409205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed860d79-cfda-4be1-a670-20eccf2324ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:32.409741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed860d79-cfda-4be1-a670-20eccf2324ca service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:32.410016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed860d79-cfda-4be1-a670-20eccf2324ca service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:32.410320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed860d79-cfda-4be1-a670-20eccf2324ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:32.411899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ed860d79-cfda-4be1-a670-20eccf2324ca service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:32.498595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed860d79-cfda-4be1-a670-20eccf2324ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:32.500176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed860d79-cfda-4be1-a670-20eccf2324ca service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:32.518337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 713/1419] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:33.639022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b35c09b-2202-45a1-96ad-840aeb223320 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:33.639598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b35c09b-2202-45a1-96ad-840aeb223320 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:33.639598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b35c09b-2202-45a1-96ad-840aeb223320 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:33.640875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 707/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:48:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:33.647066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e98d70-733d-4bd5-a29a-82ea5147fe8c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:33.647477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e98d70-733d-4bd5-a29a-82ea5147fe8c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:33.647590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e98d70-733d-4bd5-a29a-82ea5147fe8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:33.700398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:33.723973 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-60e98d70-733d-4bd5-a29a-82ea5147fe8c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:33.725167 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-60e98d70-733d-4bd5-a29a-82ea5147fe8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:33.819927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 714/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:48:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:33.970259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b35c09b-2202-45a1-96ad-840aeb223320 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:33.971677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6b35c09b-2202-45a1-96ad-840aeb223320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:34.117143 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b35c09b-2202-45a1-96ad-840aeb223320 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:34.129580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:34.227302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033e266d-8ddd-4e97-9a85-4579e724f327 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:34.227819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e266d-8ddd-4e97-9a85-4579e724f327 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:34.228092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e266d-8ddd-4e97-9a85-4579e724f327 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:34.228359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033e266d-8ddd-4e97-9a85-4579e724f327 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:34.229945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-033e266d-8ddd-4e97-9a85-4579e724f327 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:34.316424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033e266d-8ddd-4e97-9a85-4579e724f327 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:34.317983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033e266d-8ddd-4e97-9a85-4579e724f327 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:34.335760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 708/1422] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:35.445676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c08c8-5d71-4c05-bbf4-c52e60e2a994 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:35.446208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c08c8-5d71-4c05-bbf4-c52e60e2a994 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:35.446208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749c08c8-5d71-4c05-bbf4-c52e60e2a994 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:35.447483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 715/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:48:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:35.453589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e106e5d-38eb-4177-a0fa-a5b52ec01a67 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:35.454560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e106e5d-38eb-4177-a0fa-a5b52ec01a67 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:35.454703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e106e5d-38eb-4177-a0fa-a5b52ec01a67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:35.474705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:35.530278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4e106e5d-38eb-4177-a0fa-a5b52ec01a67 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:35.531512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4e106e5d-38eb-4177-a0fa-a5b52ec01a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:35.626571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 709/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:48:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:35.777041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749c08c8-5d71-4c05-bbf4-c52e60e2a994 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:35.778668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-749c08c8-5d71-4c05-bbf4-c52e60e2a994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:35.888999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:35.925643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749c08c8-5d71-4c05-bbf4-c52e60e2a994 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:36.037054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77371204-1693-4f37-bbcc-bade15152d4e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:36.037659 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77371204-1693-4f37-bbcc-bade15152d4e service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:36.037933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77371204-1693-4f37-bbcc-bade15152d4e service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:36.038239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77371204-1693-4f37-bbcc-bade15152d4e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:36.039818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-77371204-1693-4f37-bbcc-bade15152d4e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:36.115135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:36.129162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77371204-1693-4f37-bbcc-bade15152d4e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:36.130790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77371204-1693-4f37-bbcc-bade15152d4e service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:36.148609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 716/1425] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:37.262659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50684dc6-ab4f-4f30-9b7e-2b185884d48a None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:37.263220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50684dc6-ab4f-4f30-9b7e-2b185884d48a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:37.263220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50684dc6-ab4f-4f30-9b7e-2b185884d48a None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:37.264553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 710/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:48:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:37.270859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23404b93-a5e8-4c24-97cc-9e9d847a3dbf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:37.271317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23404b93-a5e8-4c24-97cc-9e9d847a3dbf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:37.271420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23404b93-a5e8-4c24-97cc-9e9d847a3dbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:37.347155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-23404b93-a5e8-4c24-97cc-9e9d847a3dbf None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:37.348340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-23404b93-a5e8-4c24-97cc-9e9d847a3dbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:37.442031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 717/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:48:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:37.593037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50684dc6-ab4f-4f30-9b7e-2b185884d48a None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:37.594607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-50684dc6-ab4f-4f30-9b7e-2b185884d48a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:37.677539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:37.739381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50684dc6-ab4f-4f30-9b7e-2b185884d48a None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:37.848190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361aceb0-ecd2-4dfd-a259-df1e6dddd326 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:37.848790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361aceb0-ecd2-4dfd-a259-df1e6dddd326 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:37.849064 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361aceb0-ecd2-4dfd-a259-df1e6dddd326 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:37.849334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361aceb0-ecd2-4dfd-a259-df1e6dddd326 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:37.850944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-361aceb0-ecd2-4dfd-a259-df1e6dddd326 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:37.900180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:37.941608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361aceb0-ecd2-4dfd-a259-df1e6dddd326 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:37.943170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361aceb0-ecd2-4dfd-a259-df1e6dddd326 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:37.961054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 711/1428] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:39.015322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a76d6f-904d-4923-bf2f-3b49edd081c8 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:39.015847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a76d6f-904d-4923-bf2f-3b49edd081c8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:39.015971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a76d6f-904d-4923-bf2f-3b49edd081c8 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:39.017287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 718/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:48:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:39.023734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbbb8ad-2bac-4abc-ba64-5d942ee56038 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:39.024194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbbb8ad-2bac-4abc-ba64-5d942ee56038 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:39.024297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbbb8ad-2bac-4abc-ba64-5d942ee56038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:39.099415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-3dbbb8ad-2bac-4abc-ba64-5d942ee56038 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:39.100611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3dbbb8ad-2bac-4abc-ba64-5d942ee56038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:39.197792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 712/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:48:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:39.349124 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a76d6f-904d-4923-bf2f-3b49edd081c8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:39.350672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-71a76d6f-904d-4923-bf2f-3b49edd081c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:39.398669 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:39.497003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a76d6f-904d-4923-bf2f-3b49edd081c8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:39.606597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c90ea0-e800-443b-a411-eb599d6a80e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:39.607118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c90ea0-e800-443b-a411-eb599d6a80e4 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:39.607393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c90ea0-e800-443b-a411-eb599d6a80e4 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:39.607662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c90ea0-e800-443b-a411-eb599d6a80e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:39.609242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-04c90ea0-e800-443b-a411-eb599d6a80e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:39.616486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:39.696196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c90ea0-e800-443b-a411-eb599d6a80e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:39.697774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c90ea0-e800-443b-a411-eb599d6a80e4 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:39.715592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 719/1431] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:40.805604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b28b95-3f74-4ee0-8315-48a98966388b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:40.806023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b28b95-3f74-4ee0-8315-48a98966388b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:40.806190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b28b95-3f74-4ee0-8315-48a98966388b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:40.807519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 713/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:48:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:40.814773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90024b8d-6844-46b6-8031-38ea2ffd7204 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:40.815338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90024b8d-6844-46b6-8031-38ea2ffd7204 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:40.815338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90024b8d-6844-46b6-8031-38ea2ffd7204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:40.893315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-90024b8d-6844-46b6-8031-38ea2ffd7204 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:40.894551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-90024b8d-6844-46b6-8031-38ea2ffd7204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:40.961422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:40.995084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 720/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:48:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:48:41.147737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b28b95-3f74-4ee0-8315-48a98966388b None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:41.149256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-52b28b95-3f74-4ee0-8315-48a98966388b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:41.166494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:41.294911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b28b95-3f74-4ee0-8315-48a98966388b None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:41.382134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:41.408186 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fe099e-c845-4e16-ac10-fa1252134a32 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:41.408790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fe099e-c845-4e16-ac10-fa1252134a32 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:41.409067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fe099e-c845-4e16-ac10-fa1252134a32 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:41.409334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fe099e-c845-4e16-ac10-fa1252134a32 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:41.410956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e0fe099e-c845-4e16-ac10-fa1252134a32 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:41.496928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fe099e-c845-4e16-ac10-fa1252134a32 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:41.498810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fe099e-c845-4e16-ac10-fa1252134a32 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:41.516030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 714/1434] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:42.581678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f329a3-b73a-44a8-aee6-b6265dc25c21 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:42.582227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f329a3-b73a-44a8-aee6-b6265dc25c21 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:42.582227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f329a3-b73a-44a8-aee6-b6265dc25c21 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:42.583573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 721/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:48:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:42.589524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5763e0-a449-41d9-a376-14bf4b1f5741 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:42.589958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5763e0-a449-41d9-a376-14bf4b1f5741 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:42.590096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5763e0-a449-41d9-a376-14bf4b1f5741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:42.663961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-cb5763e0-a449-41d9-a376-14bf4b1f5741 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:42.665150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cb5763e0-a449-41d9-a376-14bf4b1f5741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:42.711132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:42.767728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 715/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:48:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:42.919678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f329a3-b73a-44a8-aee6-b6265dc25c21 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:42.921181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b6f329a3-b73a-44a8-aee6-b6265dc25c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:43.067108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f329a3-b73a-44a8-aee6-b6265dc25c21 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:43.120732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:43.178136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3271ce7-680b-4e2d-a978-5e6a4533a2c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:43.178740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3271ce7-680b-4e2d-a978-5e6a4533a2c6 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:43.179029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3271ce7-680b-4e2d-a978-5e6a4533a2c6 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:43.179286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3271ce7-680b-4e2d-a978-5e6a4533a2c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:43.180860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c3271ce7-680b-4e2d-a978-5e6a4533a2c6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:43.268294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3271ce7-680b-4e2d-a978-5e6a4533a2c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:43.269857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3271ce7-680b-4e2d-a978-5e6a4533a2c6 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:43.287585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 722/1437] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:44.352285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee6c20a-4348-469b-9afb-dfc3d1e76227 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:44.352808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee6c20a-4348-469b-9afb-dfc3d1e76227 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:44.352808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee6c20a-4348-469b-9afb-dfc3d1e76227 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:44.354088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 716/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:48:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:44.360864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5986d6-c7c5-4861-a77c-dc2d58fcc586 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:44.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5986d6-c7c5-4861-a77c-dc2d58fcc586 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:44.361413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5986d6-c7c5-4861-a77c-dc2d58fcc586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:44.440651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8b5986d6-c7c5-4861-a77c-dc2d58fcc586 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:44.441842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8b5986d6-c7c5-4861-a77c-dc2d58fcc586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:44.443851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:44.536707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 723/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:48:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:44.686558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee6c20a-4348-469b-9afb-dfc3d1e76227 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:44.688503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-aee6c20a-4348-469b-9afb-dfc3d1e76227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:44.832201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee6c20a-4348-469b-9afb-dfc3d1e76227 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:44.858964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:44.944248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6589772a-9395-4518-b122-0a54358efc83 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:44.944777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6589772a-9395-4518-b122-0a54358efc83 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:44.945057 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6589772a-9395-4518-b122-0a54358efc83 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:44.945333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6589772a-9395-4518-b122-0a54358efc83 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:44.946962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6589772a-9395-4518-b122-0a54358efc83 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:45.036003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6589772a-9395-4518-b122-0a54358efc83 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:45.037587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6589772a-9395-4518-b122-0a54358efc83 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:45.055552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 717/1440] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:47.137406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841b4273-cce7-4432-ac9d-a8c1f480e102 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:47.137979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841b4273-cce7-4432-ac9d-a8c1f480e102 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:47.137979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841b4273-cce7-4432-ac9d-a8c1f480e102 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:47.139318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 724/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:48:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:47.144561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fab577c-a8ea-4dd8-b998-eb7569d4820d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:47.144959 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fab577c-a8ea-4dd8-b998-eb7569d4820d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:47.145119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fab577c-a8ea-4dd8-b998-eb7569d4820d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:47.185933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:47.211813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9fab577c-a8ea-4dd8-b998-eb7569d4820d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:47.213008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9fab577c-a8ea-4dd8-b998-eb7569d4820d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:47.308727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 718/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:48:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:47.593450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841b4273-cce7-4432-ac9d-a8c1f480e102 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:47.595608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-841b4273-cce7-4432-ac9d-a8c1f480e102 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:47.696579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841b4273-cce7-4432-ac9d-a8c1f480e102 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:47.728246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:47.769364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52b967b-1862-4f9d-9126-45f08d0f108b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:47.769954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52b967b-1862-4f9d-9126-45f08d0f108b service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:47.770258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52b967b-1862-4f9d-9126-45f08d0f108b service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:47.770527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52b967b-1862-4f9d-9126-45f08d0f108b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:47.772089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a52b967b-1862-4f9d-9126-45f08d0f108b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:47.878337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52b967b-1862-4f9d-9126-45f08d0f108b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:47.879682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52b967b-1862-4f9d-9126-45f08d0f108b service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:47.897233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 725/1443] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:50.191514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faaa838-69eb-4d59-acf7-81975b490cb8 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:50.192093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faaa838-69eb-4d59-acf7-81975b490cb8 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:50.192093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faaa838-69eb-4d59-acf7-81975b490cb8 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:50.193475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 719/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:50.198759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e80736-5603-4127-af92-6e72b66d0859 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:50.199159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e80736-5603-4127-af92-6e72b66d0859 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:50.199293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e80736-5603-4127-af92-6e72b66d0859 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:50.263445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d8e80736-5603-4127-af92-6e72b66d0859 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:50.264638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d8e80736-5603-4127-af92-6e72b66d0859 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:50.277453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:50.359383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 726/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:48:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:50.642146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faaa838-69eb-4d59-acf7-81975b490cb8 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:50.643673 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7faaa838-69eb-4d59-acf7-81975b490cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:50.743649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faaa838-69eb-4d59-acf7-81975b490cb8 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:50.756880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:50.818602 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5c1560-995e-496e-9559-6e23cca90967 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:50.819199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5c1560-995e-496e-9559-6e23cca90967 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:50.819478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5c1560-995e-496e-9559-6e23cca90967 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:50.819753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5c1560-995e-496e-9559-6e23cca90967 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:50.821333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0a5c1560-995e-496e-9559-6e23cca90967 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:50.929173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5c1560-995e-496e-9559-6e23cca90967 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:50.930567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5c1560-995e-496e-9559-6e23cca90967 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:50.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 720/1446] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:53.390732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92796c0b-8701-49a8-bf80-a2634adabb3c None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:53.391778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92796c0b-8701-49a8-bf80-a2634adabb3c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:53.391778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92796c0b-8701-49a8-bf80-a2634adabb3c None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:53.393081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 727/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:48:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:53.398226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb9c4bf-dc50-448c-9255-8a2232190306 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:53.398666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb9c4bf-dc50-448c-9255-8a2232190306 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:53.398807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb9c4bf-dc50-448c-9255-8a2232190306 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:53.446114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:53.464322 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-7bb9c4bf-dc50-448c-9255-8a2232190306 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:53.465524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7bb9c4bf-dc50-448c-9255-8a2232190306 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:53.563153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 721/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:48:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:48:53.848833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92796c0b-8701-49a8-bf80-a2634adabb3c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:53.850413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-92796c0b-8701-49a8-bf80-a2634adabb3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:53.950784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92796c0b-8701-49a8-bf80-a2634adabb3c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:53.990424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:54.024435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8840e8c8-afac-488b-a9ec-13d5d6418edf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:54.025466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8840e8c8-afac-488b-a9ec-13d5d6418edf service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:54.025703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8840e8c8-afac-488b-a9ec-13d5d6418edf service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:54.025994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8840e8c8-afac-488b-a9ec-13d5d6418edf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:54.028094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8840e8c8-afac-488b-a9ec-13d5d6418edf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:54.135991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8840e8c8-afac-488b-a9ec-13d5d6418edf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:54.137337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8840e8c8-afac-488b-a9ec-13d5d6418edf service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:54.155162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 728/1449] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:56.497055 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad51bca-29cd-4004-81fd-611a84016ad8 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:56.497597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad51bca-29cd-4004-81fd-611a84016ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:56.497597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad51bca-29cd-4004-81fd-611a84016ad8 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:56.499286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 722/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:48:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:56.505215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cc8954-9b97-4de1-ac84-a4c3c8d2d6b0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:56.505594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cc8954-9b97-4de1-ac84-a4c3c8d2d6b0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:56.505755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cc8954-9b97-4de1-ac84-a4c3c8d2d6b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:56.570435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-36cc8954-9b97-4de1-ac84-a4c3c8d2d6b0 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:56.571632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-36cc8954-9b97-4de1-ac84-a4c3c8d2d6b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:56.591858 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:56.665754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 729/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:48:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:56.951099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad51bca-29cd-4004-81fd-611a84016ad8 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:56.952662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3ad51bca-29cd-4004-81fd-611a84016ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:57.059807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad51bca-29cd-4004-81fd-611a84016ad8 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:48:57.079971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:57.133368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e977bd-b7fd-4538-a8c6-f1f1e616e641 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:48:57.133956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e977bd-b7fd-4538-a8c6-f1f1e616e641 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:57.134257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e977bd-b7fd-4538-a8c6-f1f1e616e641 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:57.134542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e977bd-b7fd-4538-a8c6-f1f1e616e641 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:57.136093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c7e977bd-b7fd-4538-a8c6-f1f1e616e641 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:57.240905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e977bd-b7fd-4538-a8c6-f1f1e616e641 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:48:57.242363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e977bd-b7fd-4538-a8c6-f1f1e616e641 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:48:57.260253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 723/1452] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:48:59.460969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6621f8de-b65c-4bc5-bd5f-7b519d7f0fa0 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:59.461545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6621f8de-b65c-4bc5-bd5f-7b519d7f0fa0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:59.461545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6621f8de-b65c-4bc5-bd5f-7b519d7f0fa0 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:59.462923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 730/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:48:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:48:59.468148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4289963-e2dc-4a00-8804-da673fd43b88 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:48:59.468585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4289963-e2dc-4a00-8804-da673fd43b88 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:48:59.468777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4289963-e2dc-4a00-8804-da673fd43b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:48:59.531843 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:48:59.534853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e4289963-e2dc-4a00-8804-da673fd43b88 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:48:59.536687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e4289963-e2dc-4a00-8804-da673fd43b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:48:59.631151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 724/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:48:59.917903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6621f8de-b65c-4bc5-bd5f-7b519d7f0fa0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:48:59.919492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6621f8de-b65c-4bc5-bd5f-7b519d7f0fa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:00.028417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6621f8de-b65c-4bc5-bd5f-7b519d7f0fa0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:00.081238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:00.106094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1547ad58-b75d-4c59-b250-ed3e11894aed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:00.106698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1547ad58-b75d-4c59-b250-ed3e11894aed service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:00.106972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1547ad58-b75d-4c59-b250-ed3e11894aed service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:00.107235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1547ad58-b75d-4c59-b250-ed3e11894aed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:00.108812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1547ad58-b75d-4c59-b250-ed3e11894aed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:00.220162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1547ad58-b75d-4c59-b250-ed3e11894aed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:00.221534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1547ad58-b75d-4c59-b250-ed3e11894aed service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:00.239364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 731/1455] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:48:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:02.439081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c8d7a1-3ba1-4d1d-bc63-3b815a455955 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:02.439586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c8d7a1-3ba1-4d1d-bc63-3b815a455955 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:02.439586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c8d7a1-3ba1-4d1d-bc63-3b815a455955 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:02.440973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 725/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:49:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:02.446493 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcc9fec-001e-4d28-bbb2-dd1ce54aed6a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:02.446873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcc9fec-001e-4d28-bbb2-dd1ce54aed6a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:02.447038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcc9fec-001e-4d28-bbb2-dd1ce54aed6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:02.513278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2dcc9fec-001e-4d28-bbb2-dd1ce54aed6a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:02.514521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2dcc9fec-001e-4d28-bbb2-dd1ce54aed6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:02.543812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:02.610715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 732/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:02.897026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c8d7a1-3ba1-4d1d-bc63-3b815a455955 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:02.898594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-21c8d7a1-3ba1-4d1d-bc63-3b815a455955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:03.003024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c8d7a1-3ba1-4d1d-bc63-3b815a455955 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:03.030855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:03.078635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f76c10-ef0d-4168-b377-3981137bd26a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:03.079230 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f76c10-ef0d-4168-b377-3981137bd26a service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:03.079509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f76c10-ef0d-4168-b377-3981137bd26a service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:03.079776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f76c10-ef0d-4168-b377-3981137bd26a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:03.081339 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f9f76c10-ef0d-4168-b377-3981137bd26a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:03.186301 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f76c10-ef0d-4168-b377-3981137bd26a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:03.188247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f76c10-ef0d-4168-b377-3981137bd26a service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:03.205973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 726/1458] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:05.407571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27d284b-405e-45d7-992f-b8d610656177 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:05.408102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27d284b-405e-45d7-992f-b8d610656177 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:05.408102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27d284b-405e-45d7-992f-b8d610656177 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:05.409365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 733/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:49:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:05.414600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e028d-b247-45b7-80fa-9ea92c7b3c54 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:05.415005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e028d-b247-45b7-80fa-9ea92c7b3c54 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:05.415161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762e028d-b247-45b7-80fa-9ea92c7b3c54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:05.479696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-762e028d-b247-45b7-80fa-9ea92c7b3c54 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:05.480894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-762e028d-b247-45b7-80fa-9ea92c7b3c54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:05.489448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:05.577404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 727/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:49:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:05.858505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27d284b-405e-45d7-992f-b8d610656177 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:05.860653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e27d284b-405e-45d7-992f-b8d610656177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:05.961613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27d284b-405e-45d7-992f-b8d610656177 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:06.014699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:06.035190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf35b56-53c2-4f3c-98a1-45a32e07892d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:06.035338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf35b56-53c2-4f3c-98a1-45a32e07892d service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:06.035495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf35b56-53c2-4f3c-98a1-45a32e07892d service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:06.035766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf35b56-53c2-4f3c-98a1-45a32e07892d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:06.037353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-eaf35b56-53c2-4f3c-98a1-45a32e07892d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:06.144492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf35b56-53c2-4f3c-98a1-45a32e07892d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:06.145923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf35b56-53c2-4f3c-98a1-45a32e07892d service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:06.164127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 734/1461] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:08.372726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89464ed6-1110-486d-85c4-ff28cedb349c None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:08.373232 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89464ed6-1110-486d-85c4-ff28cedb349c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:08.373232 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89464ed6-1110-486d-85c4-ff28cedb349c None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:08.375465 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 728/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:49:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:08.380701 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef949138-2a1f-4859-9901-2d2db7aa33ca None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:08.381114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef949138-2a1f-4859-9901-2d2db7aa33ca None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:08.381255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef949138-2a1f-4859-9901-2d2db7aa33ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:08.446896 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ef949138-2a1f-4859-9901-2d2db7aa33ca None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:08.448100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ef949138-2a1f-4859-9901-2d2db7aa33ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:08.486631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:08.545520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 735/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:49:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:08.830958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89464ed6-1110-486d-85c4-ff28cedb349c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:08.832509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-89464ed6-1110-486d-85c4-ff28cedb349c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:08.933608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89464ed6-1110-486d-85c4-ff28cedb349c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:08.971487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:09.123916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d73975b-b112-46f1-99ec-1eb852fe6b1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:09.124425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d73975b-b112-46f1-99ec-1eb852fe6b1d service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:09.124694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d73975b-b112-46f1-99ec-1eb852fe6b1d service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:09.124981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d73975b-b112-46f1-99ec-1eb852fe6b1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:09.126560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6d73975b-b112-46f1-99ec-1eb852fe6b1d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:09.237709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d73975b-b112-46f1-99ec-1eb852fe6b1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:09.239145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d73975b-b112-46f1-99ec-1eb852fe6b1d service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:09.257608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 729/1464] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:11.445712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429eb4a1-2145-497c-8cf3-76bbfdb4214c None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:11.446308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429eb4a1-2145-497c-8cf3-76bbfdb4214c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:11.446308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429eb4a1-2145-497c-8cf3-76bbfdb4214c None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:11.447677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 736/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:49:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:11.454090 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d00d22-a71b-472c-bf4c-760a04182bd2 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:11.454552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d00d22-a71b-472c-bf4c-760a04182bd2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:11.454715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d00d22-a71b-472c-bf4c-760a04182bd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:11.519182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-73d00d22-a71b-472c-bf4c-760a04182bd2 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:11.520388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-73d00d22-a71b-472c-bf4c-760a04182bd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:11.537860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:11.614893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 730/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:49:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:49:11.899575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429eb4a1-2145-497c-8cf3-76bbfdb4214c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:11.901125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-429eb4a1-2145-497c-8cf3-76bbfdb4214c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:12.002431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429eb4a1-2145-497c-8cf3-76bbfdb4214c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:12.069129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:12.075988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3729b686-b2ca-4565-a8cb-3ad1252d019d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:12.076589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729b686-b2ca-4565-a8cb-3ad1252d019d service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:12.076863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729b686-b2ca-4565-a8cb-3ad1252d019d service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:12.077135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3729b686-b2ca-4565-a8cb-3ad1252d019d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:12.079391 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3729b686-b2ca-4565-a8cb-3ad1252d019d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:12.186688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729b686-b2ca-4565-a8cb-3ad1252d019d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:12.188576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3729b686-b2ca-4565-a8cb-3ad1252d019d service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:12.207153 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 737/1467] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:14.391089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ec04b-3c65-4ae5-8894-e0a58c8b2e98 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:14.391624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ec04b-3c65-4ae5-8894-e0a58c8b2e98 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:14.391624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274ec04b-3c65-4ae5-8894-e0a58c8b2e98 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:14.392889 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 731/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:49:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:14.398180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2393d2-7699-4283-a4ad-89bdf057e266 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:14.398630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2393d2-7699-4283-a4ad-89bdf057e266 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:14.398808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2393d2-7699-4283-a4ad-89bdf057e266 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:14.465115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4c2393d2-7699-4283-a4ad-89bdf057e266 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:14.466329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4c2393d2-7699-4283-a4ad-89bdf057e266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:14.515011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:14.562927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 738/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:49:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:14.850276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274ec04b-3c65-4ae5-8894-e0a58c8b2e98 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:14.851805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-274ec04b-3c65-4ae5-8894-e0a58c8b2e98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:14.953620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274ec04b-3c65-4ae5-8894-e0a58c8b2e98 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:15.002635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:15.028551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05306ee9-94f7-4475-b650-630d401e78bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:15.029151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05306ee9-94f7-4475-b650-630d401e78bc service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:15.029437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05306ee9-94f7-4475-b650-630d401e78bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:15.029708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05306ee9-94f7-4475-b650-630d401e78bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:15.031335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-05306ee9-94f7-4475-b650-630d401e78bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:15.136385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05306ee9-94f7-4475-b650-630d401e78bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:15.137722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05306ee9-94f7-4475-b650-630d401e78bc service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:15.155977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 732/1470] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:24.561541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=22 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:25.153737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=66 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:35.005979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcbd44a-acac-4ca6-9adc-ec7f0781f062 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:35.006526 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcbd44a-acac-4ca6-9adc-ec7f0781f062 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:35.006526 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcbd44a-acac-4ca6-9adc-ec7f0781f062 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:35.007920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 739/1471] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:49:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:35.013313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69521cb7-e5bd-4418-b833-fd17f86ff5d9 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:35.013705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69521cb7-e5bd-4418-b833-fd17f86ff5d9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:35.013883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69521cb7-e5bd-4418-b833-fd17f86ff5d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:35.079142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-69521cb7-e5bd-4418-b833-fd17f86ff5d9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:35.080345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-69521cb7-e5bd-4418-b833-fd17f86ff5d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:35.174026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 733/1472] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:49:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:35.480982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c879efaf-7133-400f-8517-688d4695c280 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:35.481265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c879efaf-7133-400f-8517-688d4695c280 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:35.481538 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c879efaf-7133-400f-8517-688d4695c280 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:35.482889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 740/1473] 10.222.0.25 () {54 vars in 962 bytes} [Sun Oct 2 00:49:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:35.488297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3717792-9b4c-42b2-bc34-4958f130bb91 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:35.488605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3717792-9b4c-42b2-bc34-4958f130bb91 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:35.488907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3717792-9b4c-42b2-bc34-4958f130bb91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:35.564543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-b3717792-9b4c-42b2-bc34-4958f130bb91 None None] MFA Rules not processed for user `30fc59f80f544cef8b3869e8d37bb341`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:35.565718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b3717792-9b4c-42b2-bc34-4958f130bb91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:35.585539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:35.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 734/1474] 10.222.0.25 () {58 vars in 1059 bytes} [Sun Oct 2 00:49:35 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:35.648112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c879efaf-7133-400f-8517-688d4695c280 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:35.649602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c879efaf-7133-400f-8517-688d4695c280 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:35.753884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c879efaf-7133-400f-8517-688d4695c280 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:35.827784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e072e4-1b20-42ae-bf3a-bc4d7d5d5f7a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc59f80f544cef8b3869e8d37bb341', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:35.828381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e072e4-1b20-42ae-bf3a-bc4d7d5d5f7a service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:35.828660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e072e4-1b20-42ae-bf3a-bc4d7d5d5f7a service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:35.828925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e072e4-1b20-42ae-bf3a-bc4d7d5d5f7a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:35.830564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d5e072e4-1b20-42ae-bf3a-bc4d7d5d5f7a service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:35.883526 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:35.936408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e072e4-1b20-42ae-bf3a-bc4d7d5d5f7a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:35.938438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e072e4-1b20-42ae-bf3a-bc4d7d5d5f7a service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:35.956712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 741/1475] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:38.180223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8e17d4-f433-408a-8062-34951d1966fd None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:38.180748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8e17d4-f433-408a-8062-34951d1966fd None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:38.180748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8e17d4-f433-408a-8062-34951d1966fd None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:38.182122 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 735/1476] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:49:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:38.187646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a222566-24cc-4291-819d-83ca8b55f33f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:38.188055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a222566-24cc-4291-819d-83ca8b55f33f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:38.188214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a222566-24cc-4291-819d-83ca8b55f33f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:38.252731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5a222566-24cc-4291-819d-83ca8b55f33f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:38.253929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5a222566-24cc-4291-819d-83ca8b55f33f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:38.348341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 742/1477] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:49:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:49:38.622760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8e17d4-f433-408a-8062-34951d1966fd None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:38.624505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-be8e17d4-f433-408a-8062-34951d1966fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:38.747541 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8e17d4-f433-408a-8062-34951d1966fd None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:38.795567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:38.837960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0429af06-6e90-4852-847c-51a30d672d76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:38.838578 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0429af06-6e90-4852-847c-51a30d672d76 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:38.838857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0429af06-6e90-4852-847c-51a30d672d76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:38.839126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0429af06-6e90-4852-847c-51a30d672d76 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:38.839748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0429af06-6e90-4852-847c-51a30d672d76 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:38.841443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: WARNING py.warnings [None req-0429af06-6e90-4852-847c-51a30d672d76 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:49:38.841443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: warnings.warn(msg) Oct 02 00:49:38.841443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: Oct 02 00:49:38.842607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0429af06-6e90-4852-847c-51a30d672d76 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:38.861359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 736/1478] 10.222.0.25 () {56 vars in 1108 bytes} [Sun Oct 2 00:49:38 2022] GET /identity/v3/projects => generated 2553 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:49:41.028925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05f4e84-96ae-4157-ad5a-c61126cd3c33 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:41.029584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05f4e84-96ae-4157-ad5a-c61126cd3c33 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:41.029584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05f4e84-96ae-4157-ad5a-c61126cd3c33 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:41.031614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 743/1479] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:49:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:41.036936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9def6ec5-f06c-4a25-9596-16c9154ab8fb None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:41.037377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9def6ec5-f06c-4a25-9596-16c9154ab8fb None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:41.037483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9def6ec5-f06c-4a25-9596-16c9154ab8fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:41.101517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9def6ec5-f06c-4a25-9596-16c9154ab8fb None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:41.102760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9def6ec5-f06c-4a25-9596-16c9154ab8fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:41.196829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 737/1480] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:49:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:41.492049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05f4e84-96ae-4157-ad5a-c61126cd3c33 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:41.493567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f05f4e84-96ae-4157-ad5a-c61126cd3c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:41.503578 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:41.636015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05f4e84-96ae-4157-ad5a-c61126cd3c33 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:41.724138 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:41.756022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6088ba46-3fb2-49b7-8472-3f1f0ac8d194 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:41.756621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6088ba46-3fb2-49b7-8472-3f1f0ac8d194 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:41.756893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6088ba46-3fb2-49b7-8472-3f1f0ac8d194 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:41.757162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6088ba46-3fb2-49b7-8472-3f1f0ac8d194 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:41.758773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6088ba46-3fb2-49b7-8472-3f1f0ac8d194 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:41.844853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6088ba46-3fb2-49b7-8472-3f1f0ac8d194 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:41.846455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6088ba46-3fb2-49b7-8472-3f1f0ac8d194 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:41.864135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 744/1481] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:44.192084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5245947-603d-4a27-b739-cdf57ec7e320 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:44.192635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5245947-603d-4a27-b739-cdf57ec7e320 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:44.192635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5245947-603d-4a27-b739-cdf57ec7e320 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:44.193967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 738/1482] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:49:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:44.199298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b872fda-f3a3-4fa4-82f0-f1fe20d0a4b6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:44.199736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b872fda-f3a3-4fa4-82f0-f1fe20d0a4b6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:44.199848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b872fda-f3a3-4fa4-82f0-f1fe20d0a4b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:44.264338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7b872fda-f3a3-4fa4-82f0-f1fe20d0a4b6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:44.266138 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b872fda-f3a3-4fa4-82f0-f1fe20d0a4b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:44.312883 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:44.361642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 745/1483] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:49:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:44.654478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5245947-603d-4a27-b739-cdf57ec7e320 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:44.656641 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f5245947-603d-4a27-b739-cdf57ec7e320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:44.679638 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:44.758534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5245947-603d-4a27-b739-cdf57ec7e320 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:44.832150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ded65b0-6e39-4bef-ac92-a564eb100022 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:44.832745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded65b0-6e39-4bef-ac92-a564eb100022 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:44.833018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded65b0-6e39-4bef-ac92-a564eb100022 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:44.833284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ded65b0-6e39-4bef-ac92-a564eb100022 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:44.834881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1ded65b0-6e39-4bef-ac92-a564eb100022 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:44.906943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:44.941467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ded65b0-6e39-4bef-ac92-a564eb100022 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:44.942907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ded65b0-6e39-4bef-ac92-a564eb100022 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:44.959932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 739/1484] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:47.156864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e2c662-f876-4fd3-bd93-4cb105e5e468 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:47.157429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e2c662-f876-4fd3-bd93-4cb105e5e468 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:47.157429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e2c662-f876-4fd3-bd93-4cb105e5e468 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:47.159269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 746/1485] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:49:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:47.164411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbb2880-ddc6-4c2f-93c7-002b378bc248 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:47.164815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbb2880-ddc6-4c2f-93c7-002b378bc248 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:47.164972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbb2880-ddc6-4c2f-93c7-002b378bc248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:47.229988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4fbb2880-ddc6-4c2f-93c7-002b378bc248 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:47.231212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4fbb2880-ddc6-4c2f-93c7-002b378bc248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:47.328628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 740/1486] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:49:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:47.618360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e2c662-f876-4fd3-bd93-4cb105e5e468 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:47.619891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c2e2c662-f876-4fd3-bd93-4cb105e5e468 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:47.723882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e2c662-f876-4fd3-bd93-4cb105e5e468 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:47.795039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:47.797068 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5888dc6-fff8-4d34-84a7-4eea84da0525 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:47.797744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5888dc6-fff8-4d34-84a7-4eea84da0525 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:47.798226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5888dc6-fff8-4d34-84a7-4eea84da0525 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:47.798665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5888dc6-fff8-4d34-84a7-4eea84da0525 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:47.800463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c5888dc6-fff8-4d34-84a7-4eea84da0525 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:47.906561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5888dc6-fff8-4d34-84a7-4eea84da0525 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:47.907915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5888dc6-fff8-4d34-84a7-4eea84da0525 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:47.925727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 747/1487] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:50.264946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92811924-adf0-4384-9586-a5a53dfb1fc9 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:50.265496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92811924-adf0-4384-9586-a5a53dfb1fc9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:50.265496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92811924-adf0-4384-9586-a5a53dfb1fc9 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:50.266985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 741/1488] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:49:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:50.272058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fc41a6-b658-45df-8778-c4baa254835c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:50.272441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fc41a6-b658-45df-8778-c4baa254835c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:50.272597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fc41a6-b658-45df-8778-c4baa254835c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:50.336434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e0fc41a6-b658-45df-8778-c4baa254835c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:50.337558 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e0fc41a6-b658-45df-8778-c4baa254835c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:50.388682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:50.433208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 748/1489] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:49:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:50.757976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92811924-adf0-4384-9586-a5a53dfb1fc9 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:50.759526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-92811924-adf0-4384-9586-a5a53dfb1fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:50.793905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:50.867341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92811924-adf0-4384-9586-a5a53dfb1fc9 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:50.938908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bea34a-1a6c-41d0-8103-c517f739677c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:50.939495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bea34a-1a6c-41d0-8103-c517f739677c service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:50.939767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bea34a-1a6c-41d0-8103-c517f739677c service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:50.940034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bea34a-1a6c-41d0-8103-c517f739677c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:50.941591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-55bea34a-1a6c-41d0-8103-c517f739677c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:51.018171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:51.046835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bea34a-1a6c-41d0-8103-c517f739677c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:51.048296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bea34a-1a6c-41d0-8103-c517f739677c service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:51.066421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 742/1490] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:49:53.366547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa6a6c-9f53-4d82-9e54-e8d594138bc6 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:53.367056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa6a6c-9f53-4d82-9e54-e8d594138bc6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:53.367056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fa6a6c-9f53-4d82-9e54-e8d594138bc6 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:53.368330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 749/1491] 10.222.0.25 () {54 vars in 927 bytes} [Sun Oct 2 00:49:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:49:53.373505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba9028e-7e78-4cc5-9321-2af9a60b73b1 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:53.374224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba9028e-7e78-4cc5-9321-2af9a60b73b1 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:53.374336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba9028e-7e78-4cc5-9321-2af9a60b73b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:53.437860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-dba9028e-7e78-4cc5-9321-2af9a60b73b1 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:49:53.439072 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-dba9028e-7e78-4cc5-9321-2af9a60b73b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:53.533511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 743/1492] 10.222.0.25 () {58 vars in 1024 bytes} [Sun Oct 2 00:49:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:49:53.827168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fa6a6c-9f53-4d82-9e54-e8d594138bc6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:49:53.828731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-63fa6a6c-9f53-4d82-9e54-e8d594138bc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:53.928702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fa6a6c-9f53-4d82-9e54-e8d594138bc6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:49:53.999651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbb0c0c-a9ff-44f0-95b7-94f8ac118266 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:49:54.000241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb0c0c-a9ff-44f0-95b7-94f8ac118266 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:49:54.000514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb0c0c-a9ff-44f0-95b7-94f8ac118266 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:49:54.000779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbb0c0c-a9ff-44f0-95b7-94f8ac118266 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:49:54.002387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7fbb0c0c-a9ff-44f0-95b7-94f8ac118266 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:49:54.011544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:49:54.108424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbb0c0c-a9ff-44f0-95b7-94f8ac118266 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:49:54.109970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbb0c0c-a9ff-44f0-95b7-94f8ac118266 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:49:54.127830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 750/1493] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:49:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:50:03.531971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=82 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:50:04.125683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=50 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:07.063515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cb7017-b6f3-44c6-b3c4-86521bbb653e None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:07.064005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cb7017-b6f3-44c6-b3c4-86521bbb653e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:07.064111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cb7017-b6f3-44c6-b3c4-86521bbb653e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:07.138759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-70cb7017-b6f3-44c6-b3c4-86521bbb653e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:51:07.139978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-70cb7017-b6f3-44c6-b3c4-86521bbb653e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:07.236273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 744/1494] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:51:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:51:07.244170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2318d605-1dee-4373-bf81-0308a8bf28ec None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:07.244652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2318d605-1dee-4373-bf81-0308a8bf28ec None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:07.244774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2318d605-1dee-4373-bf81-0308a8bf28ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:07.322147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2318d605-1dee-4373-bf81-0308a8bf28ec None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:51:07.323297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2318d605-1dee-4373-bf81-0308a8bf28ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:07.418582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 751/1495] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:51:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:51:07.423806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cb7017-b6f3-44c6-b3c4-86521bbb653e None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:51:07.424998 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-70cb7017-b6f3-44c6-b3c4-86521bbb653e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:07.436845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cb7017-b6f3-44c6-b3c4-86521bbb653e None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:51:07.498427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:07.532563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:51:07.533164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:07.533443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:07.533714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:07.534380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:51:07.534682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:51:07.535684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: WARNING py.warnings [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:51:07.535684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: warnings.warn(msg) Oct 02 00:51:07.535684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: Oct 02 00:51:07.536572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:51:07.547990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 745/1496] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:51:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:51:07.556335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2318d605-1dee-4373-bf81-0308a8bf28ec None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:51:07.565322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2318d605-1dee-4373-bf81-0308a8bf28ec None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:51:07.625658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:07.658174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999833a3-31cf-4606-96f7-a5173288347b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:51:07.658772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999833a3-31cf-4606-96f7-a5173288347b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:07.659043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999833a3-31cf-4606-96f7-a5173288347b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:07.659306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999833a3-31cf-4606-96f7-a5173288347b admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:07.659927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999833a3-31cf-4606-96f7-a5173288347b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:51:07.661624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-999833a3-31cf-4606-96f7-a5173288347b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:51:07.661624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Oct 02 00:51:07.661624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: Oct 02 00:51:07.662831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999833a3-31cf-4606-96f7-a5173288347b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:51:07.717784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 752/1497] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:51:07 2022] POST /identity/v3/projects => generated 377 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:51:07.721992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:51:07.723538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:07.757589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787fd802-bc26-469d-93f6-481cfa077567 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:51:07.848716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2afb345-4a64-413a-9c77-f46d94caa497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:51:07.849307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2afb345-4a64-413a-9c77-f46d94caa497 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:07.849582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2afb345-4a64-413a-9c77-f46d94caa497 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:07.849853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2afb345-4a64-413a-9c77-f46d94caa497 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:07.850512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2afb345-4a64-413a-9c77-f46d94caa497 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:51:07.851508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: WARNING py.warnings [None req-f2afb345-4a64-413a-9c77-f46d94caa497 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:51:07.851508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: warnings.warn(msg) Oct 02 00:51:07.851508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: Oct 02 00:51:07.852708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2afb345-4a64-413a-9c77-f46d94caa497 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:51:07.894484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 746/1498] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:51:07 2022] POST /identity/v3/users => generated 289 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:51:07.898719 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999833a3-31cf-4606-96f7-a5173288347b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:51:07.900249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-999833a3-31cf-4606-96f7-a5173288347b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:07.960536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999833a3-31cf-4606-96f7-a5173288347b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:51:08.045424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:08.056066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d09cd2-0224-4fc5-9ff3-bb7a20ea4eef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:51:08.056667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d09cd2-0224-4fc5-9ff3-bb7a20ea4eef admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:08.056940 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d09cd2-0224-4fc5-9ff3-bb7a20ea4eef admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:08.057207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d09cd2-0224-4fc5-9ff3-bb7a20ea4eef admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:08.057837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d09cd2-0224-4fc5-9ff3-bb7a20ea4eef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:51:08.059530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-f6d09cd2-0224-4fc5-9ff3-bb7a20ea4eef admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:51:08.059530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Oct 02 00:51:08.059530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: Oct 02 00:51:08.060442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d09cd2-0224-4fc5-9ff3-bb7a20ea4eef admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:51:08.076887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 753/1499] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:51:07 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:51:08.078113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2afb345-4a64-413a-9c77-f46d94caa497 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:51:08.079612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f2afb345-4a64-413a-9c77-f46d94caa497 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:08.091164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2afb345-4a64-413a-9c77-f46d94caa497 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:51:08.115704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:08.187519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2286c49d-8aa0-4ac9-94f4-70b969906c8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:51:08.188118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2286c49d-8aa0-4ac9-94f4-70b969906c8a admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:08.188389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2286c49d-8aa0-4ac9-94f4-70b969906c8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:08.188655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2286c49d-8aa0-4ac9-94f4-70b969906c8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:08.189285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2286c49d-8aa0-4ac9-94f4-70b969906c8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:51:08.190304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: WARNING py.warnings [None req-2286c49d-8aa0-4ac9-94f4-70b969906c8a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:51:08.190304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: warnings.warn(msg) Oct 02 00:51:08.190304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: Oct 02 00:51:08.191185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2286c49d-8aa0-4ac9-94f4-70b969906c8a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:51:08.203096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 747/1500] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:51:08 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:51:08.207379 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d09cd2-0224-4fc5-9ff3-bb7a20ea4eef admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:51:08.208939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f6d09cd2-0224-4fc5-9ff3-bb7a20ea4eef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:08.243349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d09cd2-0224-4fc5-9ff3-bb7a20ea4eef admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:51:08.335340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d33c45-9b72-4494-8c15-a5fc01408015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:51:08.336001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d33c45-9b72-4494-8c15-a5fc01408015 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:08.336281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d33c45-9b72-4494-8c15-a5fc01408015 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:08.336549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d33c45-9b72-4494-8c15-a5fc01408015 admin admin] PATH_INFO: `/v3/projects/fa752b4b64cd4ad68c2baa801a2db36f/users/e967909c3359451896455374cef5942d/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:08.359698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d33c45-9b72-4494-8c15-a5fc01408015 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa752b4b64cd4ad68c2baa801a2db36f, user_id=e967909c3359451896455374cef5942d, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:51:08.360711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-03d33c45-9b72-4494-8c15-a5fc01408015 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:51:08.360711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Oct 02 00:51:08.360711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: Oct 02 00:51:08.362666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d33c45-9b72-4494-8c15-a5fc01408015 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:51:08.386595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 754/1501] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:51:08 2022] PUT /identity/v3/projects/fa752b4b64cd4ad68c2baa801a2db36f/users/e967909c3359451896455374cef5942d/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:51:08.398737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ec71ce-1aff-4d9c-bd09-d45135b57028 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:08.398959 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ec71ce-1aff-4d9c-bd09-d45135b57028 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:08.399234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ec71ce-1aff-4d9c-bd09-d45135b57028 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:08.456278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-14ec71ce-1aff-4d9c-bd09-d45135b57028 None None] MFA Rules not processed for user `e967909c3359451896455374cef5942d`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:51:08.457480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-14ec71ce-1aff-4d9c-bd09-d45135b57028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:08.533824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 748/1502] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:51:08 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:51:08.546984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91b9575-f284-4a0d-8e96-b8b6b41b0198 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:08.547271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91b9575-f284-4a0d-8e96-b8b6b41b0198 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:08.547548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91b9575-f284-4a0d-8e96-b8b6b41b0198 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:08.548923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 755/1503] 10.222.0.25 () {54 vars in 963 bytes} [Sun Oct 2 00:51:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:51:08.553506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04ccb49-eb55-4c2d-bba1-551780a4c0be None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:08.553795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04ccb49-eb55-4c2d-bba1-551780a4c0be None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:08.554150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04ccb49-eb55-4c2d-bba1-551780a4c0be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:08.586165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:08.630669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a04ccb49-eb55-4c2d-bba1-551780a4c0be None None] MFA Rules not processed for user `eb325d3c277b45c2a5c4e0e94078be29`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:51:08.631858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a04ccb49-eb55-4c2d-bba1-551780a4c0be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:08.705272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 749/1504] 10.222.0.25 () {58 vars in 1060 bytes} [Sun Oct 2 00:51:08 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:51:08.712721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91b9575-f284-4a0d-8e96-b8b6b41b0198 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:51:08.714331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b91b9575-f284-4a0d-8e96-b8b6b41b0198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:08.765078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:08.818562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91b9575-f284-4a0d-8e96-b8b6b41b0198 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:51:08.890311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38aa9528-ae67-49a9-b332-e4ad8f3aa102 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:51:08.890904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aa9528-ae67-49a9-b332-e4ad8f3aa102 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:08.891178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aa9528-ae67-49a9-b332-e4ad8f3aa102 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:08.891445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38aa9528-ae67-49a9-b332-e4ad8f3aa102 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:08.893020 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-38aa9528-ae67-49a9-b332-e4ad8f3aa102 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:08.988639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38aa9528-ae67-49a9-b332-e4ad8f3aa102 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:51:08.989984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38aa9528-ae67-49a9-b332-e4ad8f3aa102 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:51:08.993392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:09.008532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 756/1505] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:51:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:51:09.054642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa72078-b814-4b85-ae08-c0da2be90320 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:09.054932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa72078-b814-4b85-ae08-c0da2be90320 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:09.055210 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa72078-b814-4b85-ae08-c0da2be90320 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:09.056446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 750/1506] 10.222.0.25 () {52 vars in 830 bytes} [Sun Oct 2 00:51:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:51:09.060456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38aa9528-ae67-49a9-b332-e4ad8f3aa102 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:51:09.061644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-38aa9528-ae67-49a9-b332-e4ad8f3aa102 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:09.156482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38aa9528-ae67-49a9-b332-e4ad8f3aa102 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:51:09.247963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:51:09.248583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:09.249106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:09.249368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] PATH_INFO: `/v3/users/e967909c3359451896455374cef5942d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:09.262990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e967909c3359451896455374cef5942d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:51:09.264038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:51:09.264038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Oct 02 00:51:09.264038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: Oct 02 00:51:09.265312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:51:09.275289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:09.331267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e967909c3359451896455374cef5942d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:51:09.345150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e967909c3359451896455374cef5942d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:51:09.345693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e967909c3359451896455374cef5942d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:51:09.346135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e967909c3359451896455374cef5942d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:51:09.355103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e967909c3359451896455374cef5942d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:51:09.384139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 757/1507] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:51:09 2022] DELETE /identity/v3/users/e967909c3359451896455374cef5942d => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:51:09.388954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b24e7e-e7ce-4151-aa6f-70bd4bb3f78b None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:09.389401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b24e7e-e7ce-4151-aa6f-70bd4bb3f78b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:09.389515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b24e7e-e7ce-4151-aa6f-70bd4bb3f78b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:09.445738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e0b24e7e-e7ce-4151-aa6f-70bd4bb3f78b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:51:09.447088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e0b24e7e-e7ce-4151-aa6f-70bd4bb3f78b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:09.522334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:09.534835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 751/1508] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:51:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:51:09.544860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:51:09.546101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:09.642198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2713eef3-c30e-47c9-9ae0-301206f19f9c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:51:09.694979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:09.714730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac3f649-2a47-4304-a98c-783c0fee0654 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:51:09.715864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac3f649-2a47-4304-a98c-783c0fee0654 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:09.716139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac3f649-2a47-4304-a98c-783c0fee0654 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:09.716408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac3f649-2a47-4304-a98c-783c0fee0654 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:09.717972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2ac3f649-2a47-4304-a98c-783c0fee0654 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:09.823979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac3f649-2a47-4304-a98c-783c0fee0654 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:51:09.825358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac3f649-2a47-4304-a98c-783c0fee0654 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:51:09.843948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 758/1509] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:51:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:51:10.169148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b24e7e-e7ce-4151-aa6f-70bd4bb3f78b None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:51:10.170438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e0b24e7e-e7ce-4151-aa6f-70bd4bb3f78b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:51:10.183405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b24e7e-e7ce-4151-aa6f-70bd4bb3f78b None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:51:10.279507 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:51:10.280128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:51:10.280401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:51:10.280670 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] PATH_INFO: `/v3/projects/fa752b4b64cd4ad68c2baa801a2db36f` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:51:10.290305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa752b4b64cd4ad68c2baa801a2db36f)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:51:10.291315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: WARNING py.warnings [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:51:10.291315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: warnings.warn(msg) Oct 02 00:51:10.291315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: Oct 02 00:51:10.292509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:51:10.350913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa752b4b64cd4ad68c2baa801a2db36f', 'request_id': 'req-9b4ad932-9e88-445a-8814-18fe38c09e57'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:51:10.360036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa752b4b64cd4ad68c2baa801a2db36f', 'request_id': 'req-9b4ad932-9e88-445a-8814-18fe38c09e57'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:51:10.373134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa752b4b64cd4ad68c2baa801a2db36f', 'request_id': 'req-9b4ad932-9e88-445a-8814-18fe38c09e57'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:51:10.395514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] The token cache is being invalidate because project fa752b4b64cd4ad68c2baa801a2db36f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:51:10.396108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:51:10.396216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9b4ad932-9e88-445a-8814-18fe38c09e57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:51:10.397574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 752/1510] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:51:10 2022] DELETE /identity/v3/projects/fa752b4b64cd4ad68c2baa801a2db36f => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:51:19.841702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=62 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:51:20.393496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:52:36.475416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f32f49a-4a41-4f6a-b44b-ff4051584732 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:36.475977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f32f49a-4a41-4f6a-b44b-ff4051584732 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:36.475977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f32f49a-4a41-4f6a-b44b-ff4051584732 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:36.477764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 759/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:52:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:52:36.482491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a97eb0-50b3-4759-ab16-9e12bac252f5 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:36.482899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a97eb0-50b3-4759-ab16-9e12bac252f5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:36.483051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a97eb0-50b3-4759-ab16-9e12bac252f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:36.562507 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-c4a97eb0-50b3-4759-ab16-9e12bac252f5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:52:36.563732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c4a97eb0-50b3-4759-ab16-9e12bac252f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:52:36.668292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 753/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:52:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:52:36.820618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f32f49a-4a41-4f6a-b44b-ff4051584732 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:52:36.823357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1f32f49a-4a41-4f6a-b44b-ff4051584732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:52:36.940396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f32f49a-4a41-4f6a-b44b-ff4051584732 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:52:36.945840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaf199b-25f8-4fe2-93d7-b46f2061b396 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:52:36.947028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaf199b-25f8-4fe2-93d7-b46f2061b396 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:36.947304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaf199b-25f8-4fe2-93d7-b46f2061b396 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:36.947574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaf199b-25f8-4fe2-93d7-b46f2061b396 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:36.955438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaf199b-25f8-4fe2-93d7-b46f2061b396 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:52:36.956999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaf199b-25f8-4fe2-93d7-b46f2061b396 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:52:36.966787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 760/1513] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:52:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:52:37.993151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bce15a-30d8-4515-ae6b-692a3c62de30 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:37.993693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bce15a-30d8-4515-ae6b-692a3c62de30 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:37.993796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bce15a-30d8-4515-ae6b-692a3c62de30 None None] PATH_INFO: `/v3` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:37.995133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 754/1514] 10.222.0.43 () {54 vars in 925 bytes} [Sun Oct 2 00:52:37 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:52:38.001751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f794f4ef-b91e-4a1f-a6a6-792085bea720 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:38.002183 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f794f4ef-b91e-4a1f-a6a6-792085bea720 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:38.002318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f794f4ef-b91e-4a1f-a6a6-792085bea720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:38.058425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ff50f7-a3f3-4b69-9371-a0ec9c3928fd None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:38.058714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ff50f7-a3f3-4b69-9371-a0ec9c3928fd None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:38.059000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ff50f7-a3f3-4b69-9371-a0ec9c3928fd None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:38.060352 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 755/1515] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:52:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:52:38.066642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f794f4ef-b91e-4a1f-a6a6-792085bea720 None None] MFA Rules not processed for user `5eeff4f989ae4338a182bfdadd3a8b63`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:52:38.067021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e735c7-ba44-49fa-8ad1-7fab263e981a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:38.067310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e735c7-ba44-49fa-8ad1-7fab263e981a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:38.067586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e735c7-ba44-49fa-8ad1-7fab263e981a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:38.067853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f794f4ef-b91e-4a1f-a6a6-792085bea720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:52:38.089041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f3e735c7-ba44-49fa-8ad1-7fab263e981a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:52:38.090345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f3e735c7-ba44-49fa-8ad1-7fab263e981a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:52:38.098054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 756/1516] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:52:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:52:38.137709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:52:38.144125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 761/1517] 10.222.0.43 () {58 vars in 1015 bytes} [Sun Oct 2 00:52:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:52:38.251836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e735c7-ba44-49fa-8ad1-7fab263e981a None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:52:38.264218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e735c7-ba44-49fa-8ad1-7fab263e981a None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:52:38.269587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a283b83-e8ad-47ee-bd7d-c32778ef9c73 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:52:38.270213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a283b83-e8ad-47ee-bd7d-c32778ef9c73 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:38.270489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a283b83-e8ad-47ee-bd7d-c32778ef9c73 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:38.270758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a283b83-e8ad-47ee-bd7d-c32778ef9c73 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:38.278941 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a283b83-e8ad-47ee-bd7d-c32778ef9c73 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:52:38.280519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a283b83-e8ad-47ee-bd7d-c32778ef9c73 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:52:38.290020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 757/1518] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:52:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:52:47.034279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d6fca0-19e3-465d-bb3c-10a0316efe38 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:47.034778 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d6fca0-19e3-465d-bb3c-10a0316efe38 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:47.034987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d6fca0-19e3-465d-bb3c-10a0316efe38 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:47.036331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 762/1519] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 2 00:52:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:52:47.041609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79a8875-35ef-4fd4-8d1b-bc4bc80b5b4a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:47.042005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79a8875-35ef-4fd4-8d1b-bc4bc80b5b4a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:47.042181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79a8875-35ef-4fd4-8d1b-bc4bc80b5b4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:47.061837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f79a8875-35ef-4fd4-8d1b-bc4bc80b5b4a None None] MFA Rules not processed for user `5eeff4f989ae4338a182bfdadd3a8b63`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:52:47.063055 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f79a8875-35ef-4fd4-8d1b-bc4bc80b5b4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:52:47.071427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 758/1520] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 2 00:52:47 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:52:47.078793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d6fca0-19e3-465d-bb3c-10a0316efe38 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:52:47.091810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d6fca0-19e3-465d-bb3c-10a0316efe38 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:52:47.096697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ca7c96-79b0-4e8d-8689-32f14ffb47f7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:52:47.097282 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ca7c96-79b0-4e8d-8689-32f14ffb47f7 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:47.097591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ca7c96-79b0-4e8d-8689-32f14ffb47f7 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:47.097864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ca7c96-79b0-4e8d-8689-32f14ffb47f7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:47.106377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ca7c96-79b0-4e8d-8689-32f14ffb47f7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:52:47.107955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ca7c96-79b0-4e8d-8689-32f14ffb47f7 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:52:47.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 763/1521] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:52:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:52:49.726679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fce008-e790-42d5-adc6-9df4180a0184 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:49.727260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fce008-e790-42d5-adc6-9df4180a0184 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:49.727260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fce008-e790-42d5-adc6-9df4180a0184 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:49.729080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 759/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:52:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:52:49.734998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93ed5bd-4d99-4276-97ba-9184fa035044 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:49.735433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93ed5bd-4d99-4276-97ba-9184fa035044 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:49.735540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d93ed5bd-4d99-4276-97ba-9184fa035044 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:49.754651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d93ed5bd-4d99-4276-97ba-9184fa035044 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:52:49.755825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d93ed5bd-4d99-4276-97ba-9184fa035044 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:52:49.763572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 764/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:52:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:52:49.916730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fce008-e790-42d5-adc6-9df4180a0184 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:52:49.930945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fce008-e790-42d5-adc6-9df4180a0184 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:52:49.936454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a2a59f-c8b8-46f6-9ea0-e15ecd738444 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:52:49.937048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a2a59f-c8b8-46f6-9ea0-e15ecd738444 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:49.937330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a2a59f-c8b8-46f6-9ea0-e15ecd738444 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:49.937596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a2a59f-c8b8-46f6-9ea0-e15ecd738444 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:49.945531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a2a59f-c8b8-46f6-9ea0-e15ecd738444 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:52:49.947645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a2a59f-c8b8-46f6-9ea0-e15ecd738444 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:52:49.956663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 760/1524] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:52:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:52:51.038421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d5d5eb-bdef-4eff-b708-e7fd2a8ed192 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:51.038918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d5d5eb-bdef-4eff-b708-e7fd2a8ed192 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:51.039025 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d5d5eb-bdef-4eff-b708-e7fd2a8ed192 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:51.040311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 765/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 2 00:52:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:52:51.047839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3bc45-f4e2-4517-9225-5514d7fab182 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:51.048336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3bc45-f4e2-4517-9225-5514d7fab182 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:51.048436 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b3bc45-f4e2-4517-9225-5514d7fab182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:51.067198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-32b3bc45-f4e2-4517-9225-5514d7fab182 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:52:51.068404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-32b3bc45-f4e2-4517-9225-5514d7fab182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:52:51.076013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 761/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 2 00:52:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:52:51.227534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d5d5eb-bdef-4eff-b708-e7fd2a8ed192 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:52:51.242399 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d5d5eb-bdef-4eff-b708-e7fd2a8ed192 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:52:51.247800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6edd06c-58a0-44d2-b6b9-8df437ebd410 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:52:51.248637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6edd06c-58a0-44d2-b6b9-8df437ebd410 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:51.248912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6edd06c-58a0-44d2-b6b9-8df437ebd410 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:51.249180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6edd06c-58a0-44d2-b6b9-8df437ebd410 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:51.257100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6edd06c-58a0-44d2-b6b9-8df437ebd410 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:52:51.258708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6edd06c-58a0-44d2-b6b9-8df437ebd410 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:52:51.267807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 766/1527] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:52:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:52:55.167134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae79dee-a096-4519-80da-ce41886e5899 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:55.167681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae79dee-a096-4519-80da-ce41886e5899 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:55.167681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae79dee-a096-4519-80da-ce41886e5899 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:55.168986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 762/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:52:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:52:55.174231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e9c5ed-b2d2-4be4-b394-806065be2a37 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:55.174634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e9c5ed-b2d2-4be4-b394-806065be2a37 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:55.174774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e9c5ed-b2d2-4be4-b394-806065be2a37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:55.193889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-67e9c5ed-b2d2-4be4-b394-806065be2a37 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:52:55.195095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-67e9c5ed-b2d2-4be4-b394-806065be2a37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:52:55.203337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 767/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:52:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:52:55.495617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae79dee-a096-4519-80da-ce41886e5899 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:52:55.497306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2ae79dee-a096-4519-80da-ce41886e5899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:52:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:52:55.579876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae79dee-a096-4519-80da-ce41886e5899 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:52:55.583423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d0d276-0527-4d9d-be12-0fbd78b62240 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:52:55.584021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0d276-0527-4d9d-be12-0fbd78b62240 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:55.584297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0d276-0527-4d9d-be12-0fbd78b62240 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:55.584567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d0d276-0527-4d9d-be12-0fbd78b62240 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:55.593203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d0d276-0527-4d9d-be12-0fbd78b62240 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:52:55.594657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d0d276-0527-4d9d-be12-0fbd78b62240 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:52:55.603762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 763/1530] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:52:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:52:57.743567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf11ed2-4feb-4d4a-9722-2294000358b2 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:57.744111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf11ed2-4feb-4d4a-9722-2294000358b2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:57.744111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf11ed2-4feb-4d4a-9722-2294000358b2 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:57.746265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 768/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:52:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:52:57.752404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2120d3b-ebb3-49f3-985d-fb4b6cc7e0f5 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:57.752853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2120d3b-ebb3-49f3-985d-fb4b6cc7e0f5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:57.752963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2120d3b-ebb3-49f3-985d-fb4b6cc7e0f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:57.773539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a2120d3b-ebb3-49f3-985d-fb4b6cc7e0f5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:52:57.775279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a2120d3b-ebb3-49f3-985d-fb4b6cc7e0f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:52:57.783067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 764/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:52:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:52:58.074455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf11ed2-4feb-4d4a-9722-2294000358b2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:52:58.086053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf11ed2-4feb-4d4a-9722-2294000358b2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:52:58.090275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea37da2a-d5ec-4abf-97af-a26afc7637fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:52:58.090866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea37da2a-d5ec-4abf-97af-a26afc7637fe service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:52:58.091142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea37da2a-d5ec-4abf-97af-a26afc7637fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:52:58.091412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea37da2a-d5ec-4abf-97af-a26afc7637fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:52:58.100162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea37da2a-d5ec-4abf-97af-a26afc7637fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:52:58.101568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea37da2a-d5ec-4abf-97af-a26afc7637fe service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:52:58.110764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 769/1533] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:52:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:53:01.849898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70d739c-e18b-4949-b20a-6bd929ff532f None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:53:01.850494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70d739c-e18b-4949-b20a-6bd929ff532f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:53:01.850494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70d739c-e18b-4949-b20a-6bd929ff532f None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:53:01.851798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 765/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:53:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:53:01.857709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e70c1-2930-4579-9458-3cb7d12c49c3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:53:01.858482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e70c1-2930-4579-9458-3cb7d12c49c3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:53:01.858756 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e70c1-2930-4579-9458-3cb7d12c49c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:53:01.878423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-848e70c1-2930-4579-9458-3cb7d12c49c3 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:53:01.879599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-848e70c1-2930-4579-9458-3cb7d12c49c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:53:01.887124 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 770/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:53:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:53:02.193083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70d739c-e18b-4949-b20a-6bd929ff532f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:53:02.194771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f70d739c-e18b-4949-b20a-6bd929ff532f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:53:02.270258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70d739c-e18b-4949-b20a-6bd929ff532f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:53:02.273913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc12cb0-e75b-488e-b290-41f3b96eebe7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:53:02.275082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc12cb0-e75b-488e-b290-41f3b96eebe7 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:53:02.275362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc12cb0-e75b-488e-b290-41f3b96eebe7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:53:02.275632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc12cb0-e75b-488e-b290-41f3b96eebe7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:53:02.284429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc12cb0-e75b-488e-b290-41f3b96eebe7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:53:02.285841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc12cb0-e75b-488e-b290-41f3b96eebe7 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:53:02.295528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 766/1536] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:53:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:53:04.439743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0886d5c-792f-4ef8-b29d-c879bcbce7cf None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:53:04.440270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0886d5c-792f-4ef8-b29d-c879bcbce7cf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:53:04.440270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0886d5c-792f-4ef8-b29d-c879bcbce7cf None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:53:04.448232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 771/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 2 00:53:04 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:53:04.453323 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e173b-6e4b-4a03-8a89-dea0e407384a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:53:04.453734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e173b-6e4b-4a03-8a89-dea0e407384a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:53:04.453847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e173b-6e4b-4a03-8a89-dea0e407384a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:53:04.472907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-058e173b-6e4b-4a03-8a89-dea0e407384a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:53:04.474116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-058e173b-6e4b-4a03-8a89-dea0e407384a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:53:04.481787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 767/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 2 00:53:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:53:04.779554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0886d5c-792f-4ef8-b29d-c879bcbce7cf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:53:04.781230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f0886d5c-792f-4ef8-b29d-c879bcbce7cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:53:04.793876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0886d5c-792f-4ef8-b29d-c879bcbce7cf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:53:04.797326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523b38fd-43e3-4ff2-912b-f61717c9b992 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:53:04.797902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523b38fd-43e3-4ff2-912b-f61717c9b992 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:53:04.798208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523b38fd-43e3-4ff2-912b-f61717c9b992 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:53:04.798477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523b38fd-43e3-4ff2-912b-f61717c9b992 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:53:04.807509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523b38fd-43e3-4ff2-912b-f61717c9b992 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:53:04.808962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523b38fd-43e3-4ff2-912b-f61717c9b992 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:53:04.818252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 772/1539] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:53:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:53:14.468128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=66 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:53:14.814077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=34 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:30.363764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18c12c1-aaeb-4f56-9ecb-9d23b0df3051 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.392665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18c12c1-aaeb-4f56-9ecb-9d23b0df3051 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.392665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18c12c1-aaeb-4f56-9ecb-9d23b0df3051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.392665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-c18c12c1-aaeb-4f56-9ecb-9d23b0df3051 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:30.392665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c18c12c1-aaeb-4f56-9ecb-9d23b0df3051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:30.404016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 768/1540] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:54:30.412769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96304ad0-e92c-4b95-ad7f-11845b0fa415 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.413378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96304ad0-e92c-4b95-ad7f-11845b0fa415 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.413477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96304ad0-e92c-4b95-ad7f-11845b0fa415 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.447561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-96304ad0-e92c-4b95-ad7f-11845b0fa415 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:30.448813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-96304ad0-e92c-4b95-ad7f-11845b0fa415 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:30.461735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 773/1541] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:30.466723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18c12c1-aaeb-4f56-9ecb-9d23b0df3051 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:30.481440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18c12c1-aaeb-4f56-9ecb-9d23b0df3051 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:30.481768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de6dbe-9c8b-47b2-95c3-c020886ad0c6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.482322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de6dbe-9c8b-47b2-95c3-c020886ad0c6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.482812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2de6dbe-9c8b-47b2-95c3-c020886ad0c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.491522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ebfdef-dc8e-4530-87a5-59d135a61c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:30.492625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ebfdef-dc8e-4530-87a5-59d135a61c6e admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.493830 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ebfdef-dc8e-4530-87a5-59d135a61c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.494476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ebfdef-dc8e-4530-87a5-59d135a61c6e admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.495383 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ebfdef-dc8e-4530-87a5-59d135a61c6e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:30.496056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ebfdef-dc8e-4530-87a5-59d135a61c6e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:30.498247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ebfdef-dc8e-4530-87a5-59d135a61c6e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:30.518198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f2de6dbe-9c8b-47b2-95c3-c020886ad0c6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:30.518561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 769/1542] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:30.522093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f2de6dbe-9c8b-47b2-95c3-c020886ad0c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:30.527832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ebfdef-dc8e-4530-87a5-59d135a61c6e admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:30.537870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 774/1543] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:54:30.546113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ebfdef-dc8e-4530-87a5-59d135a61c6e admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:30.551888 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f078ee1-d3ef-492a-9bf6-fad3a7ebd67f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.552232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f078ee1-d3ef-492a-9bf6-fad3a7ebd67f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.552433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f078ee1-d3ef-492a-9bf6-fad3a7ebd67f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.553969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290d9f8b-2dae-40b8-acca-374f50058a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:30.554630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290d9f8b-2dae-40b8-acca-374f50058a07 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.554910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290d9f8b-2dae-40b8-acca-374f50058a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.555182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290d9f8b-2dae-40b8-acca-374f50058a07 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.556096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290d9f8b-2dae-40b8-acca-374f50058a07 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:30.557134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: WARNING py.warnings [None req-290d9f8b-2dae-40b8-acca-374f50058a07 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:54:30.557134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: warnings.warn(msg) Oct 02 00:54:30.557134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: Oct 02 00:54:30.558382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290d9f8b-2dae-40b8-acca-374f50058a07 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:30.605853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-0f078ee1-d3ef-492a-9bf6-fad3a7ebd67f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:30.607292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0f078ee1-d3ef-492a-9bf6-fad3a7ebd67f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:30.612076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 770/1544] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/projects => generated 385 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:54:30.617390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290d9f8b-2dae-40b8-acca-374f50058a07 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:30.719309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290d9f8b-2dae-40b8-acca-374f50058a07 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:30.729190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8eca30-27a9-45f4-9225-9128b840c24e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:30.733390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8eca30-27a9-45f4-9225-9128b840c24e admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.733678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8eca30-27a9-45f4-9225-9128b840c24e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.733953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8eca30-27a9-45f4-9225-9128b840c24e admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.734633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8eca30-27a9-45f4-9225-9128b840c24e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:30.735709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 775/1545] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:54:30.736947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da8eca30-27a9-45f4-9225-9128b840c24e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:30.737451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b51710-ef03-40ec-ba0c-4c51ccfe3cc3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.737950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b51710-ef03-40ec-ba0c-4c51ccfe3cc3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.738482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b51710-ef03-40ec-ba0c-4c51ccfe3cc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.773709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-43b51710-ef03-40ec-ba0c-4c51ccfe3cc3 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:30.775367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-43b51710-ef03-40ec-ba0c-4c51ccfe3cc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:30.782087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 771/1546] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/users => generated 293 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:54:30.782087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8eca30-27a9-45f4-9225-9128b840c24e admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:30.793331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 776/1547] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:30.794504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b51710-ef03-40ec-ba0c-4c51ccfe3cc3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:30.797245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8eca30-27a9-45f4-9225-9128b840c24e admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:30.812452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b51710-ef03-40ec-ba0c-4c51ccfe3cc3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:30.816833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4eba46-abb9-4f40-9f77-dd887eb4f821 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:30.819301 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4eba46-abb9-4f40-9f77-dd887eb4f821 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.819848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4eba46-abb9-4f40-9f77-dd887eb4f821 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.820350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4eba46-abb9-4f40-9f77-dd887eb4f821 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.821221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4eba46-abb9-4f40-9f77-dd887eb4f821 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:30.821750 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4eba46-abb9-4f40-9f77-dd887eb4f821 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:30.824815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74276cb6-e4e2-453d-a40a-97bcfc72a27f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:30.825708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74276cb6-e4e2-453d-a40a-97bcfc72a27f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.826241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74276cb6-e4e2-453d-a40a-97bcfc72a27f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.826742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74276cb6-e4e2-453d-a40a-97bcfc72a27f admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.827613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74276cb6-e4e2-453d-a40a-97bcfc72a27f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:30.828859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4eba46-abb9-4f40-9f77-dd887eb4f821 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:30.830417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74276cb6-e4e2-453d-a40a-97bcfc72a27f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:30.857287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 777/1548] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:30 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:30.860168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9e576b-848b-41ac-b7a2-7fb07ea80a6f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.866083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 772/1549] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:30.867555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4eba46-abb9-4f40-9f77-dd887eb4f821 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:30.868145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9e576b-848b-41ac-b7a2-7fb07ea80a6f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.868641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec9e576b-848b-41ac-b7a2-7fb07ea80a6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.892919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4eba46-abb9-4f40-9f77-dd887eb4f821 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:30.901651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ec9e576b-848b-41ac-b7a2-7fb07ea80a6f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:30.904221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ec9e576b-848b-41ac-b7a2-7fb07ea80a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:30.907864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ff0331-cc53-4d41-9680-94b66c6dd30f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:30.908481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ff0331-cc53-4d41-9680-94b66c6dd30f admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.908758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ff0331-cc53-4d41-9680-94b66c6dd30f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.909024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ff0331-cc53-4d41-9680-94b66c6dd30f admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.909663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ff0331-cc53-4d41-9680-94b66c6dd30f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:30.911280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ff0331-cc53-4d41-9680-94b66c6dd30f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:30.913926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 778/1550] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:30.915178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2a7b8-10f1-4162-97d5-87cfe63c0248 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.915462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2a7b8-10f1-4162-97d5-87cfe63c0248 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.915739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2a7b8-10f1-4162-97d5-87cfe63c0248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.923605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 773/1551] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:30 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:30.925251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ff0331-cc53-4d41-9680-94b66c6dd30f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:30.935817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ff0331-cc53-4d41-9680-94b66c6dd30f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:30.936457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a0c2a7b8-10f1-4162-97d5-87cfe63c0248 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:30.937881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a0c2a7b8-10f1-4162-97d5-87cfe63c0248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:30.940496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f07b80c-f098-49d6-8d07-82145eae1899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:30.941077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f07b80c-f098-49d6-8d07-82145eae1899 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.941362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f07b80c-f098-49d6-8d07-82145eae1899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.941638 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f07b80c-f098-49d6-8d07-82145eae1899 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.942290 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f07b80c-f098-49d6-8d07-82145eae1899 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:30.944258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f07b80c-f098-49d6-8d07-82145eae1899 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:30.945996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 779/1552] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:30.947201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43ad4b5-f847-45f1-8d3c-5a2565eca07b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.947472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43ad4b5-f847-45f1-8d3c-5a2565eca07b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.947861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43ad4b5-f847-45f1-8d3c-5a2565eca07b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.968732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e43ad4b5-f847-45f1-8d3c-5a2565eca07b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:30.969993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e43ad4b5-f847-45f1-8d3c-5a2565eca07b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:30.972309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 774/1553] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/projects => generated 393 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:30.972309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec256c6-d624-4b45-bcb4-d5c1879cb78b None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:30.972309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec256c6-d624-4b45-bcb4-d5c1879cb78b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:30.972309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec256c6-d624-4b45-bcb4-d5c1879cb78b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:30.993530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8ec256c6-d624-4b45-bcb4-d5c1879cb78b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:30.994939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8ec256c6-d624-4b45-bcb4-d5c1879cb78b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:31.033323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 780/1554] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:31.039900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6295d89d-e64e-4e5f-8b20-be0075744537 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.040196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6295d89d-e64e-4e5f-8b20-be0075744537 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.040472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6295d89d-e64e-4e5f-8b20-be0075744537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.044338 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:31.059811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 775/1555] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:31.061016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6295d89d-e64e-4e5f-8b20-be0075744537 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:31.061146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1174af53-6b90-423a-bb28-72693dbc3e32 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.061387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1174af53-6b90-423a-bb28-72693dbc3e32 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.061664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1174af53-6b90-423a-bb28-72693dbc3e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.062235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6295d89d-e64e-4e5f-8b20-be0075744537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:31.071199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 781/1556] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:31.071989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6295d89d-e64e-4e5f-8b20-be0075744537 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.081815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-1174af53-6b90-423a-bb28-72693dbc3e32 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:31.083028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1174af53-6b90-423a-bb28-72693dbc3e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:31.083168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6295d89d-e64e-4e5f-8b20-be0075744537 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.088300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5a952c-157c-4aa0-8eeb-a56307028ae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.088889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5a952c-157c-4aa0-8eeb-a56307028ae1 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.089166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5a952c-157c-4aa0-8eeb-a56307028ae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.089440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5a952c-157c-4aa0-8eeb-a56307028ae1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.090096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5a952c-157c-4aa0-8eeb-a56307028ae1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:31.090399 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5a952c-157c-4aa0-8eeb-a56307028ae1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.091316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 776/1557] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:31.091463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-dd5a952c-157c-4aa0-8eeb-a56307028ae1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:54:31.091463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Oct 02 00:54:31.091463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: Oct 02 00:54:31.092051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1174af53-6b90-423a-bb28-72693dbc3e32 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.092329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5a952c-157c-4aa0-8eeb-a56307028ae1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.111147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 782/1558] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:31.111147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a765b8-3cb5-45cd-a49d-76d2899db807 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.111147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a765b8-3cb5-45cd-a49d-76d2899db807 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.111147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a765b8-3cb5-45cd-a49d-76d2899db807 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.111576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1174af53-6b90-423a-bb28-72693dbc3e32 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.111781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82427444-7e95-43d3-9b15-a68fe55103be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.112425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82427444-7e95-43d3-9b15-a68fe55103be admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.112700 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82427444-7e95-43d3-9b15-a68fe55103be admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.112982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82427444-7e95-43d3-9b15-a68fe55103be admin admin] PATH_INFO: `/v3/projects/9a3b40887e3b44478be649dfaa3f0322/users/a852bc2fda7a479e879b7803d24530a0/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.126377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e2a765b8-3cb5-45cd-a49d-76d2899db807 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:31.127289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e2a765b8-3cb5-45cd-a49d-76d2899db807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:31.128865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82427444-7e95-43d3-9b15-a68fe55103be admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a3b40887e3b44478be649dfaa3f0322, user_id=a852bc2fda7a479e879b7803d24530a0, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.129882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: WARNING py.warnings [None req-82427444-7e95-43d3-9b15-a68fe55103be admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:54:31.129882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: warnings.warn(msg) Oct 02 00:54:31.129882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: Oct 02 00:54:31.131973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82427444-7e95-43d3-9b15-a68fe55103be admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.135075 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 783/1559] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:31.135890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a765b8-3cb5-45cd-a49d-76d2899db807 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.145880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a765b8-3cb5-45cd-a49d-76d2899db807 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.149528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 777/1560] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:31 2022] PUT /identity/v3/projects/9a3b40887e3b44478be649dfaa3f0322/users/a852bc2fda7a479e879b7803d24530a0/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:31.158192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ec5662-3ecb-487f-b6ab-172d15a79335 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.158192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ec5662-3ecb-487f-b6ab-172d15a79335 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.158192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ec5662-3ecb-487f-b6ab-172d15a79335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.172976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-58ec5662-3ecb-487f-b6ab-172d15a79335 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:31.174218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-58ec5662-3ecb-487f-b6ab-172d15a79335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:31.184473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:31.215744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e176fe6-ea3c-44d8-9b1a-36aa2fdf13e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.216616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e176fe6-ea3c-44d8-9b1a-36aa2fdf13e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.216616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e176fe6-ea3c-44d8-9b1a-36aa2fdf13e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.216882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e176fe6-ea3c-44d8-9b1a-36aa2fdf13e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.217506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e176fe6-ea3c-44d8-9b1a-36aa2fdf13e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.218559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-3e176fe6-ea3c-44d8-9b1a-36aa2fdf13e2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:54:31.218559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Oct 02 00:54:31.218559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: Oct 02 00:54:31.219770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e176fe6-ea3c-44d8-9b1a-36aa2fdf13e2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.233464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 778/1561] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:31.235097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d643cc-6cb8-41ce-831b-483b80388e72 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.235419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d643cc-6cb8-41ce-831b-483b80388e72 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.235690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d643cc-6cb8-41ce-831b-483b80388e72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.245819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 784/1562] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/users => generated 297 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.245819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bfff83-8802-4116-b750-456a9ebbd7dd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.245819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bfff83-8802-4116-b750-456a9ebbd7dd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.245819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bfff83-8802-4116-b750-456a9ebbd7dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.259013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-09d643cc-6cb8-41ce-831b-483b80388e72 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:31.260209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-09d643cc-6cb8-41ce-831b-483b80388e72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:31.266391 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-70bfff83-8802-4116-b750-456a9ebbd7dd None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:31.267599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-70bfff83-8802-4116-b750-456a9ebbd7dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:31.267977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 779/1563] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:31.269222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d22d8-07bd-4d6a-a859-cd2761e8adfc None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.269510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d22d8-07bd-4d6a-a859-cd2761e8adfc None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.269782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245d22d8-07bd-4d6a-a859-cd2761e8adfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.279526 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 785/1564] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:31.280324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bfff83-8802-4116-b750-456a9ebbd7dd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.290242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-245d22d8-07bd-4d6a-a859-cd2761e8adfc None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:31.291274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bfff83-8802-4116-b750-456a9ebbd7dd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.291461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-245d22d8-07bd-4d6a-a859-cd2761e8adfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:31.295790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c53751-6c36-433a-aa2f-316dd389fd40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.296377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c53751-6c36-433a-aa2f-316dd389fd40 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.296650 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c53751-6c36-433a-aa2f-316dd389fd40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.296919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c53751-6c36-433a-aa2f-316dd389fd40 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.297566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c53751-6c36-433a-aa2f-316dd389fd40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.299399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 780/1565] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:31.299594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c53751-6c36-433a-aa2f-316dd389fd40 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.299967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245d22d8-07bd-4d6a-a859-cd2761e8adfc None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.309099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245d22d8-07bd-4d6a-a859-cd2761e8adfc None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.313790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf92058-9d6d-46ba-946b-cb6200672ea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.314416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf92058-9d6d-46ba-946b-cb6200672ea9 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.314727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf92058-9d6d-46ba-946b-cb6200672ea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.315005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebf92058-9d6d-46ba-946b-cb6200672ea9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.315642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf92058-9d6d-46ba-946b-cb6200672ea9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:31.315942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf92058-9d6d-46ba-946b-cb6200672ea9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.317555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebf92058-9d6d-46ba-946b-cb6200672ea9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.328278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 786/1566] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/projects => generated 369 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.329581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b268bb-515c-4c2c-8c30-ff164c6f694b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.329874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b268bb-515c-4c2c-8c30-ff164c6f694b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.329997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 781/1567] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:31.330193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b268bb-515c-4c2c-8c30-ff164c6f694b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.330794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf92058-9d6d-46ba-946b-cb6200672ea9 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.341974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebf92058-9d6d-46ba-946b-cb6200672ea9 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.359590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-04b268bb-515c-4c2c-8c30-ff164c6f694b None None] MFA Rules not processed for user `a852bc2fda7a479e879b7803d24530a0`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:31.360897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-04b268bb-515c-4c2c-8c30-ff164c6f694b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:31.398991 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8259ce5e-0d60-475b-bd88-5b7d6e19554b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.399542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8259ce5e-0d60-475b-bd88-5b7d6e19554b admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.399792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8259ce5e-0d60-475b-bd88-5b7d6e19554b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.400061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8259ce5e-0d60-475b-bd88-5b7d6e19554b admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.400696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8259ce5e-0d60-475b-bd88-5b7d6e19554b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:31.401002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8259ce5e-0d60-475b-bd88-5b7d6e19554b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.402676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8259ce5e-0d60-475b-bd88-5b7d6e19554b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.414816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 782/1568] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:31.415690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8259ce5e-0d60-475b-bd88-5b7d6e19554b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.424928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8259ce5e-0d60-475b-bd88-5b7d6e19554b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.429499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fece1403-2ce1-4e2d-a631-b396b6f13102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.430091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fece1403-2ce1-4e2d-a631-b396b6f13102 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.430374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fece1403-2ce1-4e2d-a631-b396b6f13102 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.430648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fece1403-2ce1-4e2d-a631-b396b6f13102 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.431266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fece1403-2ce1-4e2d-a631-b396b6f13102 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.431743 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 787/1569] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:31.432584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b268bb-515c-4c2c-8c30-ff164c6f694b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.432916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fece1403-2ce1-4e2d-a631-b396b6f13102 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.446425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 783/1570] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:31 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:31.446425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fece1403-2ce1-4e2d-a631-b396b6f13102 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.446719 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b268bb-515c-4c2c-8c30-ff164c6f694b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.449973 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4718f55d-3306-490e-907f-e626e97f84e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.450602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4718f55d-3306-490e-907f-e626e97f84e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.450907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4718f55d-3306-490e-907f-e626e97f84e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.451178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4718f55d-3306-490e-907f-e626e97f84e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.451980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4718f55d-3306-490e-907f-e626e97f84e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:31.452232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4718f55d-3306-490e-907f-e626e97f84e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.453808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4718f55d-3306-490e-907f-e626e97f84e2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.454249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fece1403-2ce1-4e2d-a631-b396b6f13102 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.460298 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196ac89-fcbc-4bc4-a50e-306939795cb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.461034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196ac89-fcbc-4bc4-a50e-306939795cb0 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.461312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196ac89-fcbc-4bc4-a50e-306939795cb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.461586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196ac89-fcbc-4bc4-a50e-306939795cb0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.462254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1196ac89-fcbc-4bc4-a50e-306939795cb0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:31.462634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1196ac89-fcbc-4bc4-a50e-306939795cb0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.464126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1196ac89-fcbc-4bc4-a50e-306939795cb0 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.477181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 788/1571] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:31.477181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4718f55d-3306-490e-907f-e626e97f84e2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.484909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 784/1572] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:31.485791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196ac89-fcbc-4bc4-a50e-306939795cb0 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.487160 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4718f55d-3306-490e-907f-e626e97f84e2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.491880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6073b84f-b788-4b71-9fd8-bed105b9abc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.492426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6073b84f-b788-4b71-9fd8-bed105b9abc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.492704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6073b84f-b788-4b71-9fd8-bed105b9abc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.492972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6073b84f-b788-4b71-9fd8-bed105b9abc8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.493598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6073b84f-b788-4b71-9fd8-bed105b9abc8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:31.493893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6073b84f-b788-4b71-9fd8-bed105b9abc8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.495494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6073b84f-b788-4b71-9fd8-bed105b9abc8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.496211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196ac89-fcbc-4bc4-a50e-306939795cb0 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.500994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88ab1ca-0662-440d-99c1-6508c2893d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.501578 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88ab1ca-0662-440d-99c1-6508c2893d22 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.501858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88ab1ca-0662-440d-99c1-6508c2893d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.502161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88ab1ca-0662-440d-99c1-6508c2893d22 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.502784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88ab1ca-0662-440d-99c1-6508c2893d22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.504693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88ab1ca-0662-440d-99c1-6508c2893d22 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.509192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 789/1573] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:31.510090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6073b84f-b788-4b71-9fd8-bed105b9abc8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.513520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:31.521443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6073b84f-b788-4b71-9fd8-bed105b9abc8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.527158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9756ea-021f-4de2-8970-7429ce1bfeb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.527579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 785/1574] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/users => generated 285 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.528488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88ab1ca-0662-440d-99c1-6508c2893d22 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.530325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9756ea-021f-4de2-8970-7429ce1bfeb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.530674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9756ea-021f-4de2-8970-7429ce1bfeb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.530974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9756ea-021f-4de2-8970-7429ce1bfeb7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.531604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9756ea-021f-4de2-8970-7429ce1bfeb7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.533688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9756ea-021f-4de2-8970-7429ce1bfeb7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.538154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88ab1ca-0662-440d-99c1-6508c2893d22 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.542958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceca54ab-889c-4bd8-9c99-912730e1b91c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.543579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca54ab-889c-4bd8-9c99-912730e1b91c admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.543858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca54ab-889c-4bd8-9c99-912730e1b91c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.544132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceca54ab-889c-4bd8-9c99-912730e1b91c admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.544744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceca54ab-889c-4bd8-9c99-912730e1b91c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.546676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceca54ab-889c-4bd8-9c99-912730e1b91c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.560316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 790/1575] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.561257 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9756ea-021f-4de2-8970-7429ce1bfeb7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.574492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 786/1576] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/projects => generated 379 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.578971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266688f-8c89-446f-95a5-5152af8054c6 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.579254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266688f-8c89-446f-95a5-5152af8054c6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.579530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266688f-8c89-446f-95a5-5152af8054c6 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.580729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 787/1577] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 2 00:54:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:54:31.581398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7266688f-8c89-446f-95a5-5152af8054c6 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.598478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:31.627278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9756ea-021f-4de2-8970-7429ce1bfeb7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.647367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7266688f-8c89-446f-95a5-5152af8054c6 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.651927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96628912-5c65-4401-bec4-8712f1e7a646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.652469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96628912-5c65-4401-bec4-8712f1e7a646 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.652805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96628912-5c65-4401-bec4-8712f1e7a646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.653205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96628912-5c65-4401-bec4-8712f1e7a646 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.655325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96628912-5c65-4401-bec4-8712f1e7a646 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.657901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96628912-5c65-4401-bec4-8712f1e7a646 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.689460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 788/1578] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/projects => generated 387 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.689679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3f0856-1f83-4508-8d4f-2535e4c62710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.689825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96628912-5c65-4401-bec4-8712f1e7a646 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.689936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3f0856-1f83-4508-8d4f-2535e4c62710 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.689936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3f0856-1f83-4508-8d4f-2535e4c62710 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.689936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3f0856-1f83-4508-8d4f-2535e4c62710 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.689936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3f0856-1f83-4508-8d4f-2535e4c62710 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.690326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3f0856-1f83-4508-8d4f-2535e4c62710 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.700244 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 791/1579] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:31 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:31.701133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3f0856-1f83-4508-8d4f-2535e4c62710 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.710906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3f0856-1f83-4508-8d4f-2535e4c62710 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.752051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96628912-5c65-4401-bec4-8712f1e7a646 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.757201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b0bb30-fd1f-4ec6-8445-77a37dcb340c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.757807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b0bb30-fd1f-4ec6-8445-77a37dcb340c admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.758104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b0bb30-fd1f-4ec6-8445-77a37dcb340c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.758381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b0bb30-fd1f-4ec6-8445-77a37dcb340c admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.759007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b0bb30-fd1f-4ec6-8445-77a37dcb340c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.760909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b0bb30-fd1f-4ec6-8445-77a37dcb340c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.767196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c311b4d-0feb-43e1-9e7c-f6d3ff1dd4a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.767794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c311b4d-0feb-43e1-9e7c-f6d3ff1dd4a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.768173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c311b4d-0feb-43e1-9e7c-f6d3ff1dd4a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.768447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c311b4d-0feb-43e1-9e7c-f6d3ff1dd4a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.769059 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c311b4d-0feb-43e1-9e7c-f6d3ff1dd4a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.770962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c311b4d-0feb-43e1-9e7c-f6d3ff1dd4a3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.786885 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 789/1580] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/projects => generated 403 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.787911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b0bb30-fd1f-4ec6-8445-77a37dcb340c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.803991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 792/1581] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/projects => generated 379 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.803991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c311b4d-0feb-43e1-9e7c-f6d3ff1dd4a3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.829361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:31.852200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b0bb30-fd1f-4ec6-8445-77a37dcb340c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.863012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c311b4d-0feb-43e1-9e7c-f6d3ff1dd4a3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.868090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c0ecbd-4a9a-4fae-8574-c29b125704a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.868684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c0ecbd-4a9a-4fae-8574-c29b125704a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.868955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c0ecbd-4a9a-4fae-8574-c29b125704a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.869221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c0ecbd-4a9a-4fae-8574-c29b125704a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.869838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c0ecbd-4a9a-4fae-8574-c29b125704a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.872022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c0ecbd-4a9a-4fae-8574-c29b125704a2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.880735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:31.894454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 793/1582] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.895457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c0ecbd-4a9a-4fae-8574-c29b125704a2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.905045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c0ecbd-4a9a-4fae-8574-c29b125704a2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.909589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1c8efe-0cf5-45bd-b6c8-1606c2fc31f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.910197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1c8efe-0cf5-45bd-b6c8-1606c2fc31f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.910476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1c8efe-0cf5-45bd-b6c8-1606c2fc31f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.910746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1c8efe-0cf5-45bd-b6c8-1606c2fc31f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.911363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1c8efe-0cf5-45bd-b6c8-1606c2fc31f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.913263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1c8efe-0cf5-45bd-b6c8-1606c2fc31f9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.914102 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23b7f78-0888-4269-b8c2-ede335a2592a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.914723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23b7f78-0888-4269-b8c2-ede335a2592a admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.915005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23b7f78-0888-4269-b8c2-ede335a2592a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.915278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23b7f78-0888-4269-b8c2-ede335a2592a admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.915918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23b7f78-0888-4269-b8c2-ede335a2592a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.917487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23b7f78-0888-4269-b8c2-ede335a2592a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.931547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 790/1583] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:31 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:31.931547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e28b75-aaa0-40e6-91c7-c5d2e003eb55 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.931547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e28b75-aaa0-40e6-91c7-c5d2e003eb55 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.931547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e28b75-aaa0-40e6-91c7-c5d2e003eb55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.941300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 794/1584] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/users => generated 290 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.941300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1c8efe-0cf5-45bd-b6c8-1606c2fc31f9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.951248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1c8efe-0cf5-45bd-b6c8-1606c2fc31f9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.955759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef94459-afa7-4140-a1aa-e880fe563df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.956342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef94459-afa7-4140-a1aa-e880fe563df7 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.956616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef94459-afa7-4140-a1aa-e880fe563df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.956885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef94459-afa7-4140-a1aa-e880fe563df7 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:31.957498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef94459-afa7-4140-a1aa-e880fe563df7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:31.959589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef94459-afa7-4140-a1aa-e880fe563df7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:31.969033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-97e28b75-aaa0-40e6-91c7-c5d2e003eb55 None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:31.970277 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-97e28b75-aaa0-40e6-91c7-c5d2e003eb55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:31.982208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 795/1585] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/users => generated 294 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:31.983030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef94459-afa7-4140-a1aa-e880fe563df7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:31.992587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef94459-afa7-4140-a1aa-e880fe563df7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:31.997504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da377c83-7f7a-4d06-b22a-b96bca0306db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:31.998149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da377c83-7f7a-4d06-b22a-b96bca0306db admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:31.998419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da377c83-7f7a-4d06-b22a-b96bca0306db admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:31.998685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da377c83-7f7a-4d06-b22a-b96bca0306db admin admin] PATH_INFO: `/v3/projects/4c22ffb83d7d41828375606fcc6da53f/users/afe98894d83f4678bb134782d90706e8/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.014557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da377c83-7f7a-4d06-b22a-b96bca0306db admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c22ffb83d7d41828375606fcc6da53f, user_id=afe98894d83f4678bb134782d90706e8, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.017399 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da377c83-7f7a-4d06-b22a-b96bca0306db admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.032466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 796/1586] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:31 2022] PUT /identity/v3/projects/4c22ffb83d7d41828375606fcc6da53f/users/afe98894d83f4678bb134782d90706e8/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:32.033464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da377c83-7f7a-4d06-b22a-b96bca0306db admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.042458 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 791/1587] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 2 00:54:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:32.043396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e28b75-aaa0-40e6-91c7-c5d2e003eb55 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.053852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e28b75-aaa0-40e6-91c7-c5d2e003eb55 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.096708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da377c83-7f7a-4d06-b22a-b96bca0306db admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.101761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2c8d3-7912-47a9-906f-db50b4f2da34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.102376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2c8d3-7912-47a9-906f-db50b4f2da34 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.102654 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2c8d3-7912-47a9-906f-db50b4f2da34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.102919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2c8d3-7912-47a9-906f-db50b4f2da34 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.103538 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f2c8d3-7912-47a9-906f-db50b4f2da34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.105461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f2c8d3-7912-47a9-906f-db50b4f2da34 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.108877 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e103dadb-a9d1-447f-9881-33a8e219aa32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.109479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e103dadb-a9d1-447f-9881-33a8e219aa32 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.109757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e103dadb-a9d1-447f-9881-33a8e219aa32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.110032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e103dadb-a9d1-447f-9881-33a8e219aa32 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.110680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e103dadb-a9d1-447f-9881-33a8e219aa32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.112566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e103dadb-a9d1-447f-9881-33a8e219aa32 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.121114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:32.127410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 797/1588] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:32 2022] POST /identity/v3/users => generated 302 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:32.128221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2c8d3-7912-47a9-906f-db50b4f2da34 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.137871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 792/1589] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:32 2022] POST /identity/v3/users => generated 290 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:32.137871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e103dadb-a9d1-447f-9881-33a8e219aa32 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2c8d3-7912-47a9-906f-db50b4f2da34 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.143856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aec0672-3599-4550-b83a-1783ec126605 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.144590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aec0672-3599-4550-b83a-1783ec126605 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.144895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e103dadb-a9d1-447f-9881-33a8e219aa32 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.145042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aec0672-3599-4550-b83a-1783ec126605 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.145323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aec0672-3599-4550-b83a-1783ec126605 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.145961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aec0672-3599-4550-b83a-1783ec126605 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.148081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aec0672-3599-4550-b83a-1783ec126605 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.150156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61413ca0-247b-4c47-b672-7be2e01f89c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.150739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61413ca0-247b-4c47-b672-7be2e01f89c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.151015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61413ca0-247b-4c47-b672-7be2e01f89c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.151283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61413ca0-247b-4c47-b672-7be2e01f89c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.151908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61413ca0-247b-4c47-b672-7be2e01f89c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.153469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61413ca0-247b-4c47-b672-7be2e01f89c0 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.158543 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 798/1590] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.159411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aec0672-3599-4550-b83a-1783ec126605 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.169462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 793/1591] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.169462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61413ca0-247b-4c47-b672-7be2e01f89c0 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.173805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aec0672-3599-4550-b83a-1783ec126605 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.177126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61413ca0-247b-4c47-b672-7be2e01f89c0 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.178947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55942c1-51a8-4a47-8fef-e2eaae763c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.179601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55942c1-51a8-4a47-8fef-e2eaae763c6d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.179903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55942c1-51a8-4a47-8fef-e2eaae763c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.180171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55942c1-51a8-4a47-8fef-e2eaae763c6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.180980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55942c1-51a8-4a47-8fef-e2eaae763c6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.182192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d06754a-c60c-4f5b-8bac-efef514794e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.182568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55942c1-51a8-4a47-8fef-e2eaae763c6d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.182781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d06754a-c60c-4f5b-8bac-efef514794e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.183060 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d06754a-c60c-4f5b-8bac-efef514794e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.183332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d06754a-c60c-4f5b-8bac-efef514794e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.183995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d06754a-c60c-4f5b-8bac-efef514794e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.185527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d06754a-c60c-4f5b-8bac-efef514794e6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.193345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 799/1592] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.194365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17232fa1-de24-459a-b9fc-ccecbdf83659 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.194641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17232fa1-de24-459a-b9fc-ccecbdf83659 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.194903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17232fa1-de24-459a-b9fc-ccecbdf83659 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.199860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 794/1593] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.199860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d06754a-c60c-4f5b-8bac-efef514794e6 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.213349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-17232fa1-de24-459a-b9fc-ccecbdf83659 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:32.214573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-17232fa1-de24-459a-b9fc-ccecbdf83659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:32.222049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 800/1594] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:32.222828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17232fa1-de24-459a-b9fc-ccecbdf83659 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.232882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17232fa1-de24-459a-b9fc-ccecbdf83659 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.237316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03993151-32c1-4e9b-ad0f-5baef5b59f22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.237898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03993151-32c1-4e9b-ad0f-5baef5b59f22 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.238204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03993151-32c1-4e9b-ad0f-5baef5b59f22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.238475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03993151-32c1-4e9b-ad0f-5baef5b59f22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.239104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03993151-32c1-4e9b-ad0f-5baef5b59f22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.240717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03993151-32c1-4e9b-ad0f-5baef5b59f22 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.250953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 801/1595] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.251729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03993151-32c1-4e9b-ad0f-5baef5b59f22 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.261240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03993151-32c1-4e9b-ad0f-5baef5b59f22 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.265671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd30ab42-67d8-496b-beba-2463c880da07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.266284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd30ab42-67d8-496b-beba-2463c880da07 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.266561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd30ab42-67d8-496b-beba-2463c880da07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.266829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd30ab42-67d8-496b-beba-2463c880da07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.267437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd30ab42-67d8-496b-beba-2463c880da07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.268977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd30ab42-67d8-496b-beba-2463c880da07 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.275692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d06754a-c60c-4f5b-8bac-efef514794e6 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.278949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 802/1596] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 27 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.279702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd30ab42-67d8-496b-beba-2463c880da07 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.281116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8eba6a-e42f-4aca-8274-65dae890caf2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.281710 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8eba6a-e42f-4aca-8274-65dae890caf2 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.281989 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8eba6a-e42f-4aca-8274-65dae890caf2 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.282295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8eba6a-e42f-4aca-8274-65dae890caf2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.290949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd30ab42-67d8-496b-beba-2463c880da07 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.295477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1e55e0-db11-4792-a33c-f3adff8d9fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.296038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1e55e0-db11-4792-a33c-f3adff8d9fe9 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.296309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1e55e0-db11-4792-a33c-f3adff8d9fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.296575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1e55e0-db11-4792-a33c-f3adff8d9fe9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.297191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1e55e0-db11-4792-a33c-f3adff8d9fe9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.298763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1e55e0-db11-4792-a33c-f3adff8d9fe9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.312988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 803/1597] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.312988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1e55e0-db11-4792-a33c-f3adff8d9fe9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.320555 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1e55e0-db11-4792-a33c-f3adff8d9fe9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.325152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1d12b9-6644-4353-9e4d-2195f0577eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.325780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1d12b9-6644-4353-9e4d-2195f0577eaf admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.326074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1d12b9-6644-4353-9e4d-2195f0577eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.326365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1d12b9-6644-4353-9e4d-2195f0577eaf admin admin] PATH_INFO: `/v3/projects/19744acd950444688b96495eba73039a/users/3572ca950d354407a707f9dd6383713e/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.341009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1d12b9-6644-4353-9e4d-2195f0577eaf admin admin] RBAC: Authorizing `identity:create_grant(project_id=19744acd950444688b96495eba73039a, user_id=3572ca950d354407a707f9dd6383713e, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.343004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8eba6a-e42f-4aca-8274-65dae890caf2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.344120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1d12b9-6644-4353-9e4d-2195f0577eaf admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.344573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd8eba6a-e42f-4aca-8274-65dae890caf2 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.347296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=13,keystone:SELECT=87 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:32.354473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 795/1598] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:32.355271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8eba6a-e42f-4aca-8274-65dae890caf2 service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.369933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 804/1599] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:32 2022] PUT /identity/v3/projects/19744acd950444688b96495eba73039a/users/3572ca950d354407a707f9dd6383713e/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:32.369933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1d12b9-6644-4353-9e4d-2195f0577eaf admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.371272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8eba6a-e42f-4aca-8274-65dae890caf2 service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.432126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d534b149-e898-4cf1-9c8e-7b1924828921 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.433439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534b149-e898-4cf1-9c8e-7b1924828921 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.433842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1d12b9-6644-4353-9e4d-2195f0577eaf admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.434625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534b149-e898-4cf1-9c8e-7b1924828921 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.435154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d534b149-e898-4cf1-9c8e-7b1924828921 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.436824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d534b149-e898-4cf1-9c8e-7b1924828921 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.438566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871e8281-1b6f-4349-98cd-a48d5b0c920b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.439158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e8281-1b6f-4349-98cd-a48d5b0c920b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.439398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d534b149-e898-4cf1-9c8e-7b1924828921 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.439515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e8281-1b6f-4349-98cd-a48d5b0c920b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.439685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e8281-1b6f-4349-98cd-a48d5b0c920b admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.440310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871e8281-1b6f-4349-98cd-a48d5b0c920b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.441872 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871e8281-1b6f-4349-98cd-a48d5b0c920b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.446764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:32.456021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 796/1600] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.457097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 805/1601] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.457271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d534b149-e898-4cf1-9c8e-7b1924828921 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.458073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871e8281-1b6f-4349-98cd-a48d5b0c920b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.471265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871e8281-1b6f-4349-98cd-a48d5b0c920b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.476434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceee4ac8-60b1-4f77-aea4-42d0abbb56f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.477605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceee4ac8-60b1-4f77-aea4-42d0abbb56f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.477889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceee4ac8-60b1-4f77-aea4-42d0abbb56f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.478195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceee4ac8-60b1-4f77-aea4-42d0abbb56f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.478827 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceee4ac8-60b1-4f77-aea4-42d0abbb56f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.480714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceee4ac8-60b1-4f77-aea4-42d0abbb56f0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.493890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 806/1602] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.494723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceee4ac8-60b1-4f77-aea4-42d0abbb56f0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.504494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceee4ac8-60b1-4f77-aea4-42d0abbb56f0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.510753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4c4c8e-d065-4e5f-8768-1afa9f2ed1f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.511361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4c4c8e-d065-4e5f-8768-1afa9f2ed1f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.511662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4c4c8e-d065-4e5f-8768-1afa9f2ed1f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.511911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4c4c8e-d065-4e5f-8768-1afa9f2ed1f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.512527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4c4c8e-d065-4e5f-8768-1afa9f2ed1f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.514144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4c4c8e-d065-4e5f-8768-1afa9f2ed1f8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.517117 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d534b149-e898-4cf1-9c8e-7b1924828921 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.521198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b1cf6d-0549-4082-b8f6-a92a967b54c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.522579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1cf6d-0549-4082-b8f6-a92a967b54c0 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.522965 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1cf6d-0549-4082-b8f6-a92a967b54c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.523286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b1cf6d-0549-4082-b8f6-a92a967b54c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.528740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 807/1603] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:32.528740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4c4c8e-d065-4e5f-8768-1afa9f2ed1f8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.536697 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b1cf6d-0549-4082-b8f6-a92a967b54c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.538567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b1cf6d-0549-4082-b8f6-a92a967b54c0 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.539135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4c4c8e-d065-4e5f-8768-1afa9f2ed1f8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.543772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d401ebe-8be8-4167-8bbc-1c10b614c8f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.544495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d401ebe-8be8-4167-8bbc-1c10b614c8f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.544836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d401ebe-8be8-4167-8bbc-1c10b614c8f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.545177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d401ebe-8be8-4167-8bbc-1c10b614c8f3 admin admin] PATH_INFO: `/v3/projects/2b5d5f73035444e3959e308137a08fe6/users/c041f6984d154f60aeb151677f1b04b8/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.549651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 797/1604] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:32.550759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc8a933-a21c-4f78-be91-88e1c0233586 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.551041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc8a933-a21c-4f78-be91-88e1c0233586 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.551349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc8a933-a21c-4f78-be91-88e1c0233586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.562735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d401ebe-8be8-4167-8bbc-1c10b614c8f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2b5d5f73035444e3959e308137a08fe6, user_id=c041f6984d154f60aeb151677f1b04b8, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.566151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d401ebe-8be8-4167-8bbc-1c10b614c8f3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.586107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 808/1605] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:32 2022] PUT /identity/v3/projects/2b5d5f73035444e3959e308137a08fe6/users/c041f6984d154f60aeb151677f1b04b8/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:32.587006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-2cc8a933-a21c-4f78-be91-88e1c0233586 None None] MFA Rules not processed for user `3572ca950d354407a707f9dd6383713e`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:32.588550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2cc8a933-a21c-4f78-be91-88e1c0233586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:32.595004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d401ebe-8be8-4167-8bbc-1c10b614c8f3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.666905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d401ebe-8be8-4167-8bbc-1c10b614c8f3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.671578 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406995dc-07d9-43c0-bc4c-8e49334ae2f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.672234 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406995dc-07d9-43c0-bc4c-8e49334ae2f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.672510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406995dc-07d9-43c0-bc4c-8e49334ae2f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.672779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406995dc-07d9-43c0-bc4c-8e49334ae2f7 admin admin] PATH_INFO: `/v3/projects/3e983e7eb1474850a440d1e022c7c5a9/users/ee548b86b214493e83a00964137c4fde/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.678370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 798/1606] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:32 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:32.678370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc8a933-a21c-4f78-be91-88e1c0233586 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.690832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc8a933-a21c-4f78-be91-88e1c0233586 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.695726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406995dc-07d9-43c0-bc4c-8e49334ae2f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e983e7eb1474850a440d1e022c7c5a9, user_id=ee548b86b214493e83a00964137c4fde, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.698727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406995dc-07d9-43c0-bc4c-8e49334ae2f7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.699378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59891b82-555d-4cc7-a476-64f152cf4b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.700404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59891b82-555d-4cc7-a476-64f152cf4b91 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.700923 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59891b82-555d-4cc7-a476-64f152cf4b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.701305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59891b82-555d-4cc7-a476-64f152cf4b91 admin admin] PATH_INFO: `/v3/projects/ff4aef28dfc444ee914b56c64b2edb77/users/a8894fd1044241ee8ba84927b6e8326a/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.712794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 809/1607] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:32 2022] PUT /identity/v3/projects/3e983e7eb1474850a440d1e022c7c5a9/users/ee548b86b214493e83a00964137c4fde/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:32.714159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406995dc-07d9-43c0-bc4c-8e49334ae2f7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.720871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59891b82-555d-4cc7-a476-64f152cf4b91 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff4aef28dfc444ee914b56c64b2edb77, user_id=a8894fd1044241ee8ba84927b6e8326a, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.724817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59891b82-555d-4cc7-a476-64f152cf4b91 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.745001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 799/1608] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:32 2022] PUT /identity/v3/projects/ff4aef28dfc444ee914b56c64b2edb77/users/a8894fd1044241ee8ba84927b6e8326a/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:32.746026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59891b82-555d-4cc7-a476-64f152cf4b91 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.753271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:32.792191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406995dc-07d9-43c0-bc4c-8e49334ae2f7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.827653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59891b82-555d-4cc7-a476-64f152cf4b91 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:32.832228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d4920a-17a5-49cc-8925-2abae11d30a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.832914 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d4920a-17a5-49cc-8925-2abae11d30a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.833186 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d4920a-17a5-49cc-8925-2abae11d30a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.833455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d4920a-17a5-49cc-8925-2abae11d30a1 admin admin] PATH_INFO: `/v3/projects/aa0e51fa2dfe4f6f8f4e4fab437cb8b8/users/7932a6409fba4ccaaddbc6e5cb93709b/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.847621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:32.853368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d4920a-17a5-49cc-8925-2abae11d30a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa0e51fa2dfe4f6f8f4e4fab437cb8b8, user_id=7932a6409fba4ccaaddbc6e5cb93709b, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.855886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31f3c2f-a39d-4baa-8193-afd4512f5981 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:32.856546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f3c2f-a39d-4baa-8193-afd4512f5981 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.856820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f3c2f-a39d-4baa-8193-afd4512f5981 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.857053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d4920a-17a5-49cc-8925-2abae11d30a1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.857176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31f3c2f-a39d-4baa-8193-afd4512f5981 admin admin] PATH_INFO: `/v3/projects/f1e48389be37426b82d4ab4533808445/users/8a95888e0a6148ae9df8c8e47789348b/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.874053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31f3c2f-a39d-4baa-8193-afd4512f5981 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1e48389be37426b82d4ab4533808445, user_id=8a95888e0a6148ae9df8c8e47789348b, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:32.874829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 800/1609] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:32 2022] PUT /identity/v3/projects/aa0e51fa2dfe4f6f8f4e4fab437cb8b8/users/7932a6409fba4ccaaddbc6e5cb93709b/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:32.876239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4629cc-0f59-4226-ad97-482fd0f0a1e6 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.876713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4629cc-0f59-4226-ad97-482fd0f0a1e6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.877667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4629cc-0f59-4226-ad97-482fd0f0a1e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.885489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31f3c2f-a39d-4baa-8193-afd4512f5981 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:32.906349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 810/1610] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:32 2022] PUT /identity/v3/projects/f1e48389be37426b82d4ab4533808445/users/8a95888e0a6148ae9df8c8e47789348b/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:32.907735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ae732f-2f5c-4eeb-a81e-de4c2b9880ab None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:32.908015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ae732f-2f5c-4eeb-a81e-de4c2b9880ab None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:32.908289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ae732f-2f5c-4eeb-a81e-de4c2b9880ab None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:32.909666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 811/1611] 10.222.0.25 () {54 vars in 912 bytes} [Sun Oct 2 00:54:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:54:32.910227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ae732f-2f5c-4eeb-a81e-de4c2b9880ab None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:32.936563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4e4629cc-0f59-4226-ad97-482fd0f0a1e6 None None] MFA Rules not processed for user `c041f6984d154f60aeb151677f1b04b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:32.938362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4e4629cc-0f59-4226-ad97-482fd0f0a1e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:33.007808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ae732f-2f5c-4eeb-a81e-de4c2b9880ab None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.012708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95aa6c4c-301b-4797-97dc-918b76053fea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.014157 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aa6c4c-301b-4797-97dc-918b76053fea admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.014535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aa6c4c-301b-4797-97dc-918b76053fea admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.015038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aa6c4c-301b-4797-97dc-918b76053fea admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.015787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95aa6c4c-301b-4797-97dc-918b76053fea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:33.019155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95aa6c4c-301b-4797-97dc-918b76053fea admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:33.022227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 801/1612] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:33.024147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6e3d82-99aa-468b-a654-57a04cbaa6a6 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.024626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6e3d82-99aa-468b-a654-57a04cbaa6a6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.025513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6e3d82-99aa-468b-a654-57a04cbaa6a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.051760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-3f6e3d82-99aa-468b-a654-57a04cbaa6a6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:33.053731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3f6e3d82-99aa-468b-a654-57a04cbaa6a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:33.062253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 812/1613] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:32 2022] POST /identity/v3/projects => generated 373 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:33.062253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489d6ffc-cf2f-4700-846b-70973df68ac1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.062253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489d6ffc-cf2f-4700-846b-70973df68ac1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.062253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489d6ffc-cf2f-4700-846b-70973df68ac1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.068573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 802/1614] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:33.068573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dcb3d8-6fe7-4ab3-9e9a-be3409a16dc6 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.068573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dcb3d8-6fe7-4ab3-9e9a-be3409a16dc6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.068948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dcb3d8-6fe7-4ab3-9e9a-be3409a16dc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.091058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-489d6ffc-cf2f-4700-846b-70973df68ac1 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:33.092379 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-489d6ffc-cf2f-4700-846b-70973df68ac1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:33.102634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f6dcb3d8-6fe7-4ab3-9e9a-be3409a16dc6 None None] MFA Rules not processed for user `7932a6409fba4ccaaddbc6e5cb93709b`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:33.103849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f6dcb3d8-6fe7-4ab3-9e9a-be3409a16dc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:33.141532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:33.164702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 813/1615] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:33.166034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3071594f-8953-48bd-b34e-919412746303 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.166874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3071594f-8953-48bd-b34e-919412746303 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.166874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3071594f-8953-48bd-b34e-919412746303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.182724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 803/1616] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:33.182724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ceafb-0f1a-4ed0-aa5a-2ebc491c59f0 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.182724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ceafb-0f1a-4ed0-aa5a-2ebc491c59f0 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.182724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ceafb-0f1a-4ed0-aa5a-2ebc491c59f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.199464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3071594f-8953-48bd-b34e-919412746303 None None] MFA Rules not processed for user `8a95888e0a6148ae9df8c8e47789348b`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:33.200687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3071594f-8953-48bd-b34e-919412746303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:33.201890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-7b2ceafb-0f1a-4ed0-aa5a-2ebc491c59f0 None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:33.203089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7b2ceafb-0f1a-4ed0-aa5a-2ebc491c59f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:33.264879 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:33.278727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 814/1617] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:33.280045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92efbe22-e858-4bd1-b023-ef66a6c23e04 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.280331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92efbe22-e858-4bd1-b023-ef66a6c23e04 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.280606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92efbe22-e858-4bd1-b023-ef66a6c23e04 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.282245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 815/1618] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 2 00:54:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:54:33.283893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92efbe22-e858-4bd1-b023-ef66a6c23e04 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.288714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 804/1619] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:33.289668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2ceafb-0f1a-4ed0-aa5a-2ebc491c59f0 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.303755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92efbe22-e858-4bd1-b023-ef66a6c23e04 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.308883 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aaeeee-99ce-4631-a4c2-33cab7fe5f97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.310119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aaeeee-99ce-4631-a4c2-33cab7fe5f97 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.310457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aaeeee-99ce-4631-a4c2-33cab7fe5f97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.310787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35aaeeee-99ce-4631-a4c2-33cab7fe5f97 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.311345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35aaeeee-99ce-4631-a4c2-33cab7fe5f97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:33.313324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35aaeeee-99ce-4631-a4c2-33cab7fe5f97 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:33.350215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 816/1620] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/users => generated 293 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:33.350215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aaeeee-99ce-4631-a4c2-33cab7fe5f97 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.375265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2ceafb-0f1a-4ed0-aa5a-2ebc491c59f0 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.381412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2620e9-6b5d-46ac-8463-06d5275c66ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.382910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2620e9-6b5d-46ac-8463-06d5275c66ba service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.383194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2620e9-6b5d-46ac-8463-06d5275c66ba service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.383760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2620e9-6b5d-46ac-8463-06d5275c66ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.399332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2620e9-6b5d-46ac-8463-06d5275c66ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:33.402086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2620e9-6b5d-46ac-8463-06d5275c66ba service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:33.417095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 805/1621] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:33.418025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2620e9-6b5d-46ac-8463-06d5275c66ba service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.418307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35aaeeee-99ce-4631-a4c2-33cab7fe5f97 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.422133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27758eef-35e0-41b2-abb8-c8d4f20ab71d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.422878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27758eef-35e0-41b2-abb8-c8d4f20ab71d service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.423422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27758eef-35e0-41b2-abb8-c8d4f20ab71d service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.423741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27758eef-35e0-41b2-abb8-c8d4f20ab71d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.428546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2620e9-6b5d-46ac-8463-06d5275c66ba service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.433447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e9c2b2-7d7e-4f7f-a2bc-55b1fb5e5da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.434126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e9c2b2-7d7e-4f7f-a2bc-55b1fb5e5da5 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.434382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e9c2b2-7d7e-4f7f-a2bc-55b1fb5e5da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.434701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e9c2b2-7d7e-4f7f-a2bc-55b1fb5e5da5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.435468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e9c2b2-7d7e-4f7f-a2bc-55b1fb5e5da5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:33.437621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e9c2b2-7d7e-4f7f-a2bc-55b1fb5e5da5 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:33.441394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27758eef-35e0-41b2-abb8-c8d4f20ab71d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:33.447082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27758eef-35e0-41b2-abb8-c8d4f20ab71d service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:33.460874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 817/1622] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:33.461967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27758eef-35e0-41b2-abb8-c8d4f20ab71d service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.476265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27758eef-35e0-41b2-abb8-c8d4f20ab71d service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.480167 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 806/1623] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/projects => generated 381 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:33.484188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729ebb46-69b2-4e67-8774-a968db067e51 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.484491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729ebb46-69b2-4e67-8774-a968db067e51 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.485556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729ebb46-69b2-4e67-8774-a968db067e51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.511604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-729ebb46-69b2-4e67-8774-a968db067e51 None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:33.512813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-729ebb46-69b2-4e67-8774-a968db067e51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:33.554594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28d3baf-40a0-4a1a-a700-a526188e1551 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.555357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28d3baf-40a0-4a1a-a700-a526188e1551 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.555736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28d3baf-40a0-4a1a-a700-a526188e1551 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.558033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28d3baf-40a0-4a1a-a700-a526188e1551 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.558033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28d3baf-40a0-4a1a-a700-a526188e1551 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:33.559338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28d3baf-40a0-4a1a-a700-a526188e1551 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:33.564777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:33.597571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 818/1624] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/projects => generated 403 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:33.598902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28d3baf-40a0-4a1a-a700-a526188e1551 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.608956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 807/1625] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:33.620394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729ebb46-69b2-4e67-8774-a968db067e51 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.633820 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729ebb46-69b2-4e67-8774-a968db067e51 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.680152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28d3baf-40a0-4a1a-a700-a526188e1551 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.684206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbf40af-2165-4fbf-97ba-e181be839b3f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.684829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbf40af-2165-4fbf-97ba-e181be839b3f service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.685104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbf40af-2165-4fbf-97ba-e181be839b3f service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.685374 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbf40af-2165-4fbf-97ba-e181be839b3f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.714961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a253fe8-b6a4-481c-b2ae-bb580cce3730 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.716067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a253fe8-b6a4-481c-b2ae-bb580cce3730 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.716443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a253fe8-b6a4-481c-b2ae-bb580cce3730 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.716826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a253fe8-b6a4-481c-b2ae-bb580cce3730 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.717783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a253fe8-b6a4-481c-b2ae-bb580cce3730 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:33.720520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a253fe8-b6a4-481c-b2ae-bb580cce3730 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:33.740488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 808/1626] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:33 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:33.741873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a253fe8-b6a4-481c-b2ae-bb580cce3730 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.749715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:33.754726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a253fe8-b6a4-481c-b2ae-bb580cce3730 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.769583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7a01f1-db57-4a45-b4d6-c4bc2440e0bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.770829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7a01f1-db57-4a45-b4d6-c4bc2440e0bc admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.771280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7a01f1-db57-4a45-b4d6-c4bc2440e0bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.772301 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7a01f1-db57-4a45-b4d6-c4bc2440e0bc admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.773051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7a01f1-db57-4a45-b4d6-c4bc2440e0bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:33.775492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7a01f1-db57-4a45-b4d6-c4bc2440e0bc admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:33.792885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbf40af-2165-4fbf-97ba-e181be839b3f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:33.794486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbf40af-2165-4fbf-97ba-e181be839b3f service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:33.814081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 819/1627] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:33.823569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 809/1628] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/users => generated 297 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:33.827506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7a01f1-db57-4a45-b4d6-c4bc2440e0bc admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.827687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbf40af-2165-4fbf-97ba-e181be839b3f service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.852898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7a01f1-db57-4a45-b4d6-c4bc2440e0bc admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.853117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbf40af-2165-4fbf-97ba-e181be839b3f service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:33.859616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3718f58d-c93b-4876-823d-5e8bc620ef36 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.860430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3718f58d-c93b-4876-823d-5e8bc620ef36 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.860823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3718f58d-c93b-4876-823d-5e8bc620ef36 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.861172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3718f58d-c93b-4876-823d-5e8bc620ef36 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.866041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a054d072-fa8e-4482-a197-8d1e1e280ffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:33.866408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a054d072-fa8e-4482-a197-8d1e1e280ffc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:33.866687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a054d072-fa8e-4482-a197-8d1e1e280ffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:33.866964 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a054d072-fa8e-4482-a197-8d1e1e280ffc admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:33.867909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a054d072-fa8e-4482-a197-8d1e1e280ffc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:33.872163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a054d072-fa8e-4482-a197-8d1e1e280ffc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:33.951668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 820/1629] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:33 2022] POST /identity/v3/users => generated 308 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:33.956147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a054d072-fa8e-4482-a197-8d1e1e280ffc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:33.992793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a054d072-fa8e-4482-a197-8d1e1e280ffc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:34.005242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5bd714-44b3-47ed-8ebd-7405436977cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:34.006950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5bd714-44b3-47ed-8ebd-7405436977cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:34.007226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5bd714-44b3-47ed-8ebd-7405436977cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:34.007497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5bd714-44b3-47ed-8ebd-7405436977cf admin admin] PATH_INFO: `/v3/projects/bef8c130f3284910ad0c635ce534b48d/users/d92f520659d94fce87044fb2f33a67bf/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:34.027433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3718f58d-c93b-4876-823d-5e8bc620ef36 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:34.029157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3718f58d-c93b-4876-823d-5e8bc620ef36 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:34.044315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5bd714-44b3-47ed-8ebd-7405436977cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=bef8c130f3284910ad0c635ce534b48d, user_id=d92f520659d94fce87044fb2f33a67bf, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:34.045204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 810/1630] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:34.047314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5bd714-44b3-47ed-8ebd-7405436977cf admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:34.060644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3718f58d-c93b-4876-823d-5e8bc620ef36 service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:34.073742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 821/1631] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:33 2022] PUT /identity/v3/projects/bef8c130f3284910ad0c635ce534b48d/users/d92f520659d94fce87044fb2f33a67bf/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:34.074730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5bd714-44b3-47ed-8ebd-7405436977cf admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:34.085009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3718f58d-c93b-4876-823d-5e8bc620ef36 service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:34.176278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:34.210239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5bd714-44b3-47ed-8ebd-7405436977cf admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:34.223158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82580f6-fd89-479f-b6df-c2129c1f7525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:34.224020 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82580f6-fd89-479f-b6df-c2129c1f7525 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:34.224154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82580f6-fd89-479f-b6df-c2129c1f7525 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:34.224372 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82580f6-fd89-479f-b6df-c2129c1f7525 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:34.225074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82580f6-fd89-479f-b6df-c2129c1f7525 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:34.226830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82580f6-fd89-479f-b6df-c2129c1f7525 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:34.230127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aaf13e-59de-4944-8d8b-01fd34621b20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:34.230827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aaf13e-59de-4944-8d8b-01fd34621b20 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:34.231048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aaf13e-59de-4944-8d8b-01fd34621b20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:34.231462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aaf13e-59de-4944-8d8b-01fd34621b20 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:34.232115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82aaf13e-59de-4944-8d8b-01fd34621b20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:34.234532 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82aaf13e-59de-4944-8d8b-01fd34621b20 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:34.253107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 822/1632] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:34 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:34.256916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82580f6-fd89-479f-b6df-c2129c1f7525 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:34.257741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 811/1633] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:34 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:34.258802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aaf13e-59de-4944-8d8b-01fd34621b20 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:34.274631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aaf13e-59de-4944-8d8b-01fd34621b20 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:34.280714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bf15fc-5c2e-4951-8a87-db9c98e0103b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:34.281151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82580f6-fd89-479f-b6df-c2129c1f7525 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:34.281386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bf15fc-5c2e-4951-8a87-db9c98e0103b admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:34.281709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bf15fc-5c2e-4951-8a87-db9c98e0103b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:34.282160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bf15fc-5c2e-4951-8a87-db9c98e0103b admin admin] PATH_INFO: `/v3/projects/492bd17cf2b045b296ec3bf5beec8bc2/users/104bcb37fcbe40a39264bb498e31a16d/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:34.289285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e1f4e8-576c-4c2c-889b-0f97cdc69340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:34.289954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e1f4e8-576c-4c2c-889b-0f97cdc69340 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:34.290277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e1f4e8-576c-4c2c-889b-0f97cdc69340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:34.290539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e1f4e8-576c-4c2c-889b-0f97cdc69340 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:34.291178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e1f4e8-576c-4c2c-889b-0f97cdc69340 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:34.293977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e1f4e8-576c-4c2c-889b-0f97cdc69340 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:34.317531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bf15fc-5c2e-4951-8a87-db9c98e0103b admin admin] RBAC: Authorizing `identity:create_grant(project_id=492bd17cf2b045b296ec3bf5beec8bc2, user_id=104bcb37fcbe40a39264bb498e31a16d, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:34.320140 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 823/1634] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:34 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:34.321053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e1f4e8-576c-4c2c-889b-0f97cdc69340 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:34.324858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bf15fc-5c2e-4951-8a87-db9c98e0103b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:34.336968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e1f4e8-576c-4c2c-889b-0f97cdc69340 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:34.351334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 812/1635] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:34 2022] PUT /identity/v3/projects/492bd17cf2b045b296ec3bf5beec8bc2/users/104bcb37fcbe40a39264bb498e31a16d/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:34.352205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bf15fc-5c2e-4951-8a87-db9c98e0103b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:34.358052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0086ab-2f14-4f1f-b260-39f8733b11c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:34.358636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0086ab-2f14-4f1f-b260-39f8733b11c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:34.358910 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0086ab-2f14-4f1f-b260-39f8733b11c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:34.359180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0086ab-2f14-4f1f-b260-39f8733b11c3 admin admin] PATH_INFO: `/v3/projects/95dd8c4315154fe4980d3f4a23477823/users/1748baf6a8854cbe82ba8524994b18ad/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:34.412848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0086ab-2f14-4f1f-b260-39f8733b11c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95dd8c4315154fe4980d3f4a23477823, user_id=1748baf6a8854cbe82ba8524994b18ad, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:34.425762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0086ab-2f14-4f1f-b260-39f8733b11c3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:34.464318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 824/1636] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:34 2022] PUT /identity/v3/projects/95dd8c4315154fe4980d3f4a23477823/users/1748baf6a8854cbe82ba8524994b18ad/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:34.466948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0086ab-2f14-4f1f-b260-39f8733b11c3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:34.573255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:34.573255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bf15fc-5c2e-4951-8a87-db9c98e0103b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:34.717349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0086ab-2f14-4f1f-b260-39f8733b11c3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:34.745636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebd816b-7488-40eb-a1cb-cdf4be487822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:34.747846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd816b-7488-40eb-a1cb-cdf4be487822 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:34.750721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd816b-7488-40eb-a1cb-cdf4be487822 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:34.760610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd816b-7488-40eb-a1cb-cdf4be487822 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:34.769696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d48aef7-e72c-4dfe-b1e1-42b1bb37ebf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:34.770428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d48aef7-e72c-4dfe-b1e1-42b1bb37ebf7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:34.770710 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d48aef7-e72c-4dfe-b1e1-42b1bb37ebf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:34.770983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d48aef7-e72c-4dfe-b1e1-42b1bb37ebf7 admin admin] PATH_INFO: `/v3/projects/bef8c130f3284910ad0c635ce534b48d/users/d92f520659d94fce87044fb2f33a67bf/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:34.771730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebd816b-7488-40eb-a1cb-cdf4be487822 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:34.773119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d48aef7-e72c-4dfe-b1e1-42b1bb37ebf7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bef8c130f3284910ad0c635ce534b48d, user_id=d92f520659d94fce87044fb2f33a67bf, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:34.776531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d48aef7-e72c-4dfe-b1e1-42b1bb37ebf7 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:34.777300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebd816b-7488-40eb-a1cb-cdf4be487822 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:34.814141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 813/1637] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:34 2022] PUT /identity/v3/projects/bef8c130f3284910ad0c635ce534b48d/users/d92f520659d94fce87044fb2f33a67bf/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:34.818499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d48aef7-e72c-4dfe-b1e1-42b1bb37ebf7 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:34.829386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 825/1638] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:34 2022] GET /identity/v3/roles => generated 1420 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:34.830865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff04c93-1b6b-4d86-ace9-ad1abe5d413d None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:34.831146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff04c93-1b6b-4d86-ace9-ad1abe5d413d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:34.831414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff04c93-1b6b-4d86-ace9-ad1abe5d413d None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:34.833596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 826/1639] 10.222.0.25 () {54 vars in 912 bytes} [Sun Oct 2 00:54:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:54:34.834819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916076f0-1af9-48c0-904e-ac2219689cd7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:34.835098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916076f0-1af9-48c0-904e-ac2219689cd7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:34.835367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916076f0-1af9-48c0-904e-ac2219689cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:34.902676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-916076f0-1af9-48c0-904e-ac2219689cd7 None None] MFA Rules not processed for user `d92f520659d94fce87044fb2f33a67bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:34.905670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-916076f0-1af9-48c0-904e-ac2219689cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:34.993597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d48aef7-e72c-4dfe-b1e1-42b1bb37ebf7 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:35.007300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e212af-2403-4f2d-b3e5-3321128aa404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:35.007931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e212af-2403-4f2d-b3e5-3321128aa404 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:35.008206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e212af-2403-4f2d-b3e5-3321128aa404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:35.048499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e212af-2403-4f2d-b3e5-3321128aa404 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:35.049289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e212af-2403-4f2d-b3e5-3321128aa404 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:35.051141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e212af-2403-4f2d-b3e5-3321128aa404 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:35.096649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 814/1640] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:34 2022] GET /identity/v3/roles => generated 1420 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:35.098984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e212af-2403-4f2d-b3e5-3321128aa404 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:35.130551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e212af-2403-4f2d-b3e5-3321128aa404 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:35.145153 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 827/1641] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:34 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:35.147657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b606554-d41d-41df-ad60-32b681f4f935 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:35.148196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b606554-d41d-41df-ad60-32b681f4f935 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:35.148695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b606554-d41d-41df-ad60-32b681f4f935 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:35.150282 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5441e6fe-c6fd-4552-b908-e4b23c5a98ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:35.150968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5441e6fe-c6fd-4552-b908-e4b23c5a98ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:35.151245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5441e6fe-c6fd-4552-b908-e4b23c5a98ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:35.151568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5441e6fe-c6fd-4552-b908-e4b23c5a98ab admin admin] PATH_INFO: `/v3/projects/492bd17cf2b045b296ec3bf5beec8bc2/users/104bcb37fcbe40a39264bb498e31a16d/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:35.157234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5441e6fe-c6fd-4552-b908-e4b23c5a98ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=492bd17cf2b045b296ec3bf5beec8bc2, user_id=104bcb37fcbe40a39264bb498e31a16d, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:35.166158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5441e6fe-c6fd-4552-b908-e4b23c5a98ab admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:35.181444 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 815/1642] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:35 2022] PUT /identity/v3/projects/492bd17cf2b045b296ec3bf5beec8bc2/users/104bcb37fcbe40a39264bb498e31a16d/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:35.182758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5441e6fe-c6fd-4552-b908-e4b23c5a98ab admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:35.185972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1b606554-d41d-41df-ad60-32b681f4f935 None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:35.187286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1b606554-d41d-41df-ad60-32b681f4f935 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:35.226594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:35.332436 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5441e6fe-c6fd-4552-b908-e4b23c5a98ab admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:35.338876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ae2eea-b6c2-407c-b1df-bcece5860e4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:35.340594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ae2eea-b6c2-407c-b1df-bcece5860e4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:35.341553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ae2eea-b6c2-407c-b1df-bcece5860e4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:35.342273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ae2eea-b6c2-407c-b1df-bcece5860e4b admin admin] PATH_INFO: `/v3/projects/95dd8c4315154fe4980d3f4a23477823/users/1748baf6a8854cbe82ba8524994b18ad/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:35.342519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 828/1643] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 00:54:35 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:35.343526 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b606554-d41d-41df-ad60-32b681f4f935 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:35.359690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ae2eea-b6c2-407c-b1df-bcece5860e4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=95dd8c4315154fe4980d3f4a23477823, user_id=1748baf6a8854cbe82ba8524994b18ad, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:35.364429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ae2eea-b6c2-407c-b1df-bcece5860e4b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:35.384974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 816/1644] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:35 2022] PUT /identity/v3/projects/95dd8c4315154fe4980d3f4a23477823/users/1748baf6a8854cbe82ba8524994b18ad/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:35.386706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c235a233-3d54-4de2-9a26-83c55ac1e83c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:35.387573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c235a233-3d54-4de2-9a26-83c55ac1e83c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:35.388068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c235a233-3d54-4de2-9a26-83c55ac1e83c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:35.446113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-c235a233-3d54-4de2-9a26-83c55ac1e83c None None] MFA Rules not processed for user `104bcb37fcbe40a39264bb498e31a16d`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:35.449926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c235a233-3d54-4de2-9a26-83c55ac1e83c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:35.457758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:35.483069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b606554-d41d-41df-ad60-32b681f4f935 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:35.592902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7071b84d-1a54-4c08-85c3-9262767b35b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:35.593524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7071b84d-1a54-4c08-85c3-9262767b35b4 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:35.593789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7071b84d-1a54-4c08-85c3-9262767b35b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:35.594052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7071b84d-1a54-4c08-85c3-9262767b35b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:35.616723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 817/1645] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:35 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:35.619567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c235a233-3d54-4de2-9a26-83c55ac1e83c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:35.685834 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c235a233-3d54-4de2-9a26-83c55ac1e83c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:35.697051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecba1d43-52b8-40d1-906d-c9bae8a2ec0a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:35.697991 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba1d43-52b8-40d1-906d-c9bae8a2ec0a service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:35.698318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba1d43-52b8-40d1-906d-c9bae8a2ec0a service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:35.698875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecba1d43-52b8-40d1-906d-c9bae8a2ec0a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:35.748308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7071b84d-1a54-4c08-85c3-9262767b35b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:35.749840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7071b84d-1a54-4c08-85c3-9262767b35b4 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:35.772407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 829/1646] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:35 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:35.773728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afefcec0-21b0-4bcc-95fd-a8fba2fbaf4f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:35.774012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afefcec0-21b0-4bcc-95fd-a8fba2fbaf4f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:35.774646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afefcec0-21b0-4bcc-95fd-a8fba2fbaf4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:35.839180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-afefcec0-21b0-4bcc-95fd-a8fba2fbaf4f None None] MFA Rules not processed for user `1748baf6a8854cbe82ba8524994b18ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:35.840422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-afefcec0-21b0-4bcc-95fd-a8fba2fbaf4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:35.883974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:35.929540 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecba1d43-52b8-40d1-906d-c9bae8a2ec0a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:35.929540 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecba1d43-52b8-40d1-906d-c9bae8a2ec0a service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:35.962034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 818/1647] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:35.963033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecba1d43-52b8-40d1-906d-c9bae8a2ec0a service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:35.983289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecba1d43-52b8-40d1-906d-c9bae8a2ec0a service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:35.987753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b998e39-6485-4f79-b739-9439e7b53769 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:35.988205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b998e39-6485-4f79-b739-9439e7b53769 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:35.988485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b998e39-6485-4f79-b739-9439e7b53769 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:35.988758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b998e39-6485-4f79-b739-9439e7b53769 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:35.991017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 830/1648] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:35 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:36.010418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b998e39-6485-4f79-b739-9439e7b53769 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:36.012870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b998e39-6485-4f79-b739-9439e7b53769 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:36.037760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 819/1649] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:35 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:36.228836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afefcec0-21b0-4bcc-95fd-a8fba2fbaf4f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:36.243779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afefcec0-21b0-4bcc-95fd-a8fba2fbaf4f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:36.261532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bc0593-bb5c-4f1d-ae9a-b4d4df79761f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:36.262335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bc0593-bb5c-4f1d-ae9a-b4d4df79761f service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:36.262759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bc0593-bb5c-4f1d-ae9a-b4d4df79761f service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:36.263084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06bc0593-bb5c-4f1d-ae9a-b4d4df79761f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:36.283845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bc0593-bb5c-4f1d-ae9a-b4d4df79761f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:36.285503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06bc0593-bb5c-4f1d-ae9a-b4d4df79761f service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:36.310093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 831/1650] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:36 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:36.486882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b998e39-6485-4f79-b739-9439e7b53769 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:36.496573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b998e39-6485-4f79-b739-9439e7b53769 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:36.512011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aea045b-37ef-4c6c-add6-2161c478ca5d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:36.512780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aea045b-37ef-4c6c-add6-2161c478ca5d service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:36.513260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aea045b-37ef-4c6c-add6-2161c478ca5d service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:36.513527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aea045b-37ef-4c6c-add6-2161c478ca5d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:36.652863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aea045b-37ef-4c6c-add6-2161c478ca5d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:36.655243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aea045b-37ef-4c6c-add6-2161c478ca5d service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:36.686000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 820/1651] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:36 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:37.111755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bc0593-bb5c-4f1d-ae9a-b4d4df79761f service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:37.123328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06bc0593-bb5c-4f1d-ae9a-b4d4df79761f service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:37.136954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a90f20-9cbd-4165-92e4-150bb5f0bbd2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:37.137588 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a90f20-9cbd-4165-92e4-150bb5f0bbd2 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:37.137861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a90f20-9cbd-4165-92e4-150bb5f0bbd2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:37.138177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a90f20-9cbd-4165-92e4-150bb5f0bbd2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:37.317262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a90f20-9cbd-4165-92e4-150bb5f0bbd2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:37.321156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a90f20-9cbd-4165-92e4-150bb5f0bbd2 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:37.348527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 832/1652] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:37 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:37.764852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aea045b-37ef-4c6c-add6-2161c478ca5d service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:37.804286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:38.026400 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aea045b-37ef-4c6c-add6-2161c478ca5d service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:38.045693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:38.046588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:38.048248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:38.048687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] PATH_INFO: `/v3/users/8a95888e0a6148ae9df8c8e47789348b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:38.082908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a95888e0a6148ae9df8c8e47789348b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:38.090720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: WARNING py.warnings [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:54:38.090720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: warnings.warn(msg) Oct 02 00:54:38.090720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: Oct 02 00:54:38.098344 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:38.307906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a95888e0a6148ae9df8c8e47789348b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:38.310940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a95888e0a6148ae9df8c8e47789348b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:38.330586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a95888e0a6148ae9df8c8e47789348b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:38.331795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a95888e0a6148ae9df8c8e47789348b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:38.350201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a95888e0a6148ae9df8c8e47789348b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:38.398776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 821/1653] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:54:37 2022] DELETE /identity/v3/users/8a95888e0a6148ae9df8c8e47789348b => generated 0 bytes in 630 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:38.399328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96f9f8c-777b-41ea-8fbd-714560742d08 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:38.400717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96f9f8c-777b-41ea-8fbd-714560742d08 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:38.401278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96f9f8c-777b-41ea-8fbd-714560742d08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:38.453150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a96f9f8c-777b-41ea-8fbd-714560742d08 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:38.454728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a96f9f8c-777b-41ea-8fbd-714560742d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:38.688873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 833/1654] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:54:38.705761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:38.708280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:38.912982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360c746d-4a80-4007-b7d4-98e3490b058e admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:38.929075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75fdb79-d845-48a0-bb68-50104686925c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:38.933678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75fdb79-d845-48a0-bb68-50104686925c service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:38.936277 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75fdb79-d845-48a0-bb68-50104686925c service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:38.938001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75fdb79-d845-48a0-bb68-50104686925c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:38.980765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75fdb79-d845-48a0-bb68-50104686925c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:38.984063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75fdb79-d845-48a0-bb68-50104686925c service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:38.998485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 822/1655] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:41.225195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96f9f8c-777b-41ea-8fbd-714560742d08 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:41.226874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a96f9f8c-777b-41ea-8fbd-714560742d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:41.287608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96f9f8c-777b-41ea-8fbd-714560742d08 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:41.298990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:41.300011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:41.300381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:41.301490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] PATH_INFO: `/v3/projects/f1e48389be37426b82d4ab4533808445` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:41.302914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1e48389be37426b82d4ab4533808445)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:41.312290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: WARNING py.warnings [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 02 00:54:41.312290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: warnings.warn(msg) Oct 02 00:54:41.312290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: Oct 02 00:54:41.319245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:41.446879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:41.520341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1e48389be37426b82d4ab4533808445', 'request_id': 'req-df49a61d-d715-42a4-9fd6-7849fbf71818'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:41.549451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1e48389be37426b82d4ab4533808445', 'request_id': 'req-df49a61d-d715-42a4-9fd6-7849fbf71818'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:41.601702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1e48389be37426b82d4ab4533808445', 'request_id': 'req-df49a61d-d715-42a4-9fd6-7849fbf71818'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:41.645293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] The token cache is being invalidate because project f1e48389be37426b82d4ab4533808445 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:54:41.645751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:41.646614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df49a61d-d715-42a4-9fd6-7849fbf71818 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:41.652458 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 834/1656] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:54:41 2022] DELETE /identity/v3/projects/f1e48389be37426b82d4ab4533808445 => generated 0 bytes in 426 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:41.654284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75fdb79-d845-48a0-bb68-50104686925c service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:41.679495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75fdb79-d845-48a0-bb68-50104686925c service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:41.733581 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:41.819780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:41.822484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:41.822774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:41.824727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] PATH_INFO: `/v3/users/104bcb37fcbe40a39264bb498e31a16d` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:41.828437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] RBAC: Authorizing `identity:delete_user(user_id=104bcb37fcbe40a39264bb498e31a16d)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:41.835337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:41.952677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '104bcb37fcbe40a39264bb498e31a16d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:41.953440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '104bcb37fcbe40a39264bb498e31a16d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:41.969894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '104bcb37fcbe40a39264bb498e31a16d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:41.971130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '104bcb37fcbe40a39264bb498e31a16d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:42.006544 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '104bcb37fcbe40a39264bb498e31a16d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:42.058046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 823/1657] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:54:41 2022] DELETE /identity/v3/users/104bcb37fcbe40a39264bb498e31a16d => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:42.606908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abea8e-d463-45af-9a92-ed3746f80b76 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:42.615991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abea8e-d463-45af-9a92-ed3746f80b76 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:42.615991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abea8e-d463-45af-9a92-ed3746f80b76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:42.640123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:42.641574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:42.709531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-69abea8e-d463-45af-9a92-ed3746f80b76 None None] MFA Rules not processed for user `ee548b86b214493e83a00964137c4fde`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:42.712315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-69abea8e-d463-45af-9a92-ed3746f80b76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:42.867864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0c81f3-744d-424f-953d-bb056363a99c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:42.878757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:42.879651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:42.880111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:42.880337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] PATH_INFO: `/v3/users/c041f6984d154f60aeb151677f1b04b8` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:42.882593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c041f6984d154f60aeb151677f1b04b8)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:42.884845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:42.977157 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 835/1658] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:42 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 371 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:42.978652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a49d3-cf73-4d89-afa5-c7b06505c992 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:42.978936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a49d3-cf73-4d89-afa5-c7b06505c992 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:42.980540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a49d3-cf73-4d89-afa5-c7b06505c992 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:43.009747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c041f6984d154f60aeb151677f1b04b8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:43.010587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c041f6984d154f60aeb151677f1b04b8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:43.017530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:43.038836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-076a49d3-cf73-4d89-afa5-c7b06505c992 None None] MFA Rules not processed for user `ee548b86b214493e83a00964137c4fde`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:43.039777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c041f6984d154f60aeb151677f1b04b8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:43.040906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-076a49d3-cf73-4d89-afa5-c7b06505c992 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:43.042896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c041f6984d154f60aeb151677f1b04b8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:43.063112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-521bab1a-3408-4bdf-a1cd-67bf47eacc41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c041f6984d154f60aeb151677f1b04b8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:43.071518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 836/1659] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:42 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:43.072758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a49d3-cf73-4d89-afa5-c7b06505c992 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:43.078907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-076a49d3-cf73-4d89-afa5-c7b06505c992 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:43.109962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 824/1660] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:54:42 2022] DELETE /identity/v3/users/c041f6984d154f60aeb151677f1b04b8 => generated 0 bytes in 470 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:43.114495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4261648-4977-41cc-ab3e-b33aec2cd195 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:43.114939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4261648-4977-41cc-ab3e-b33aec2cd195 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:43.115788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4261648-4977-41cc-ab3e-b33aec2cd195 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:43.141948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-c4261648-4977-41cc-ab3e-b33aec2cd195 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:43.143335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c4261648-4977-41cc-ab3e-b33aec2cd195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:43.211538 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a49d3-cf73-4d89-afa5-c7b06505c992 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:43.274575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:43.276228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 825/1661] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:43.303587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4261648-4977-41cc-ab3e-b33aec2cd195 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:43.304464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c4261648-4977-41cc-ab3e-b33aec2cd195 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:43.317823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d869c13-6196-48e3-ac59-662f3714065f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:43.319214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d869c13-6196-48e3-ac59-662f3714065f service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:43.320562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d869c13-6196-48e3-ac59-662f3714065f service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:43.320562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d869c13-6196-48e3-ac59-662f3714065f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:43.324461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9d869c13-6196-48e3-ac59-662f3714065f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:43.469952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4261648-4977-41cc-ab3e-b33aec2cd195 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:43.475735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d869c13-6196-48e3-ac59-662f3714065f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:43.478111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d869c13-6196-48e3-ac59-662f3714065f service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:43.483695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76be7e64-cd82-4797-8c95-93f945729c95 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:43.484340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76be7e64-cd82-4797-8c95-93f945729c95 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:43.484622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76be7e64-cd82-4797-8c95-93f945729c95 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:43.484905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76be7e64-cd82-4797-8c95-93f945729c95 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:43.502764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 837/1662] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:43 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:43.509739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76be7e64-cd82-4797-8c95-93f945729c95 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:43.512222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76be7e64-cd82-4797-8c95-93f945729c95 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:43.525765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d869c13-6196-48e3-ac59-662f3714065f service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:43.529923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9d869c13-6196-48e3-ac59-662f3714065f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:43.542013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 826/1663] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:54:43.598627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d869c13-6196-48e3-ac59-662f3714065f service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:43.614343 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:43.615852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:43.616329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:43.616823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] PATH_INFO: `/v3/projects/492bd17cf2b045b296ec3bf5beec8bc2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:43.620920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] RBAC: Authorizing `identity:delete_project(project_id=492bd17cf2b045b296ec3bf5beec8bc2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:43.625026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:43.783243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '492bd17cf2b045b296ec3bf5beec8bc2', 'request_id': 'req-fd0b8e51-49e2-4e97-81a1-87dca855f10a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:43.807483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '492bd17cf2b045b296ec3bf5beec8bc2', 'request_id': 'req-fd0b8e51-49e2-4e97-81a1-87dca855f10a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:43.815567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '492bd17cf2b045b296ec3bf5beec8bc2', 'request_id': 'req-fd0b8e51-49e2-4e97-81a1-87dca855f10a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:43.828436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] The token cache is being invalidate because project 492bd17cf2b045b296ec3bf5beec8bc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:54:43.829067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:43.829953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fd0b8e51-49e2-4e97-81a1-87dca855f10a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:43.831988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 838/1664] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:54:43 2022] DELETE /identity/v3/projects/492bd17cf2b045b296ec3bf5beec8bc2 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:43.837245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061154c6-b5f7-454d-8d98-0a16d7974fc5 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:43.837705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061154c6-b5f7-454d-8d98-0a16d7974fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:43.838647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061154c6-b5f7-454d-8d98-0a16d7974fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:43.861262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-061154c6-b5f7-454d-8d98-0a16d7974fc5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:43.863057 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-061154c6-b5f7-454d-8d98-0a16d7974fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:43.991748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 827/1665] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:44.001369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1540b9f3-3a01-41ad-bf2e-2dfe173a6d2b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.001654 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1540b9f3-3a01-41ad-bf2e-2dfe173a6d2b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.001929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1540b9f3-3a01-41ad-bf2e-2dfe173a6d2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.043065 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1540b9f3-3a01-41ad-bf2e-2dfe173a6d2b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:44.051545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1540b9f3-3a01-41ad-bf2e-2dfe173a6d2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:44.073614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 839/1666] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:44.077768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061154c6-b5f7-454d-8d98-0a16d7974fc5 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:44.090234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061154c6-b5f7-454d-8d98-0a16d7974fc5 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:44.097737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18722457-b750-4b46-9e79-8f7c92af788d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:44.098625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18722457-b750-4b46-9e79-8f7c92af788d admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.099056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18722457-b750-4b46-9e79-8f7c92af788d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.099513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18722457-b750-4b46-9e79-8f7c92af788d admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.100620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18722457-b750-4b46-9e79-8f7c92af788d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:44.101111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18722457-b750-4b46-9e79-8f7c92af788d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:44.105207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18722457-b750-4b46-9e79-8f7c92af788d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:44.122250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 828/1667] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:44.127338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1540b9f3-3a01-41ad-bf2e-2dfe173a6d2b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:44.140605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1540b9f3-3a01-41ad-bf2e-2dfe173a6d2b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:44.146147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cca6475-8d1e-46fb-aa94-ddaac9351634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:44.147206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cca6475-8d1e-46fb-aa94-ddaac9351634 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.147678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cca6475-8d1e-46fb-aa94-ddaac9351634 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.148110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cca6475-8d1e-46fb-aa94-ddaac9351634 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.148943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cca6475-8d1e-46fb-aa94-ddaac9351634 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:44.151537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cca6475-8d1e-46fb-aa94-ddaac9351634 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:44.214517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 840/1668] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:44 2022] POST /identity/v3/projects => generated 367 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:54:44.217546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18722457-b750-4b46-9e79-8f7c92af788d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:44.225275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:44.233085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18722457-b750-4b46-9e79-8f7c92af788d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:44.311455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea9901d-eb41-4724-86e8-ddd7c7cf8954 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.312067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea9901d-eb41-4724-86e8-ddd7c7cf8954 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.312568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea9901d-eb41-4724-86e8-ddd7c7cf8954 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.363200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4732f6bd-70fd-41f9-900a-0f584db8cead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:44.363381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4732f6bd-70fd-41f9-900a-0f584db8cead admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.363602 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4732f6bd-70fd-41f9-900a-0f584db8cead admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.364058 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4732f6bd-70fd-41f9-900a-0f584db8cead admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.364756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4732f6bd-70fd-41f9-900a-0f584db8cead admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:44.368901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4732f6bd-70fd-41f9-900a-0f584db8cead admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:44.386233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fea9901d-eb41-4724-86e8-ddd7c7cf8954 None None] MFA Rules not processed for user `afe98894d83f4678bb134782d90706e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:44.387516 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fea9901d-eb41-4724-86e8-ddd7c7cf8954 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:44.404413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 829/1669] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:44 2022] POST /identity/v3/users => generated 284 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:44.409401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32c5217-e92d-4ab9-be83-252275f3a6d0 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.409690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32c5217-e92d-4ab9-be83-252275f3a6d0 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.411141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32c5217-e92d-4ab9-be83-252275f3a6d0 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.418696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 830/1670] 10.222.0.25 () {54 vars in 912 bytes} [Sun Oct 2 00:54:44 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:54:44.418696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32c5217-e92d-4ab9-be83-252275f3a6d0 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:44.434532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:44.443394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32c5217-e92d-4ab9-be83-252275f3a6d0 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:44.455261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee7a324-ba21-4459-82e7-cb29d89a360d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:44.456893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee7a324-ba21-4459-82e7-cb29d89a360d admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.457355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee7a324-ba21-4459-82e7-cb29d89a360d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.457760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee7a324-ba21-4459-82e7-cb29d89a360d admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.458722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee7a324-ba21-4459-82e7-cb29d89a360d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:44.460645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee7a324-ba21-4459-82e7-cb29d89a360d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:44.478144 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 831/1671] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:44 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:44.479718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12796c3c-1e79-4532-adfe-0ea8217bd27a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.480127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12796c3c-1e79-4532-adfe-0ea8217bd27a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.480534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12796c3c-1e79-4532-adfe-0ea8217bd27a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.595496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 841/1672] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:44 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:44.596985 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea9901d-eb41-4724-86e8-ddd7c7cf8954 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:44.597790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-12796c3c-1e79-4532-adfe-0ea8217bd27a None None] MFA Rules not processed for user `e75811ee9ed7490985fed336ab9feb19`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:44.605459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-12796c3c-1e79-4532-adfe-0ea8217bd27a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:44.611969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea9901d-eb41-4724-86e8-ddd7c7cf8954 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:44.627659 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b0fb82-d49c-4506-9c19-22eff840cf33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:44.628946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b0fb82-d49c-4506-9c19-22eff840cf33 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.629360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b0fb82-d49c-4506-9c19-22eff840cf33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.629762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b0fb82-d49c-4506-9c19-22eff840cf33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.631536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b0fb82-d49c-4506-9c19-22eff840cf33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:44.633487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b0fb82-d49c-4506-9c19-22eff840cf33 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:44.659122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 842/1673] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:44 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:44.660264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b0fb82-d49c-4506-9c19-22eff840cf33 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:44.665133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-17b0fb82-d49c-4506-9c19-22eff840cf33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:44.700506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b0fb82-d49c-4506-9c19-22eff840cf33 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:44.716980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:44.717627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.717900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.718195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] PATH_INFO: `/v3/projects/2b5d5f73035444e3959e308137a08fe6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.723479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2b5d5f73035444e3959e308137a08fe6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:44.726051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:44.763869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 832/1674] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 00:54:44 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:44.765901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0871d7e-a8e7-4449-b069-24bcd4021a30 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:44.766542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0871d7e-a8e7-4449-b069-24bcd4021a30 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:44.767044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0871d7e-a8e7-4449-b069-24bcd4021a30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:44.840019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f0871d7e-a8e7-4449-b069-24bcd4021a30 None None] MFA Rules not processed for user `afe98894d83f4678bb134782d90706e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:44.840019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f0871d7e-a8e7-4449-b069-24bcd4021a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:44.888211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2b5d5f73035444e3959e308137a08fe6', 'request_id': 'req-d80b4b46-ffb7-448c-a82b-08efa2517a7c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:44.907976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2b5d5f73035444e3959e308137a08fe6', 'request_id': 'req-d80b4b46-ffb7-448c-a82b-08efa2517a7c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:44.924456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2b5d5f73035444e3959e308137a08fe6', 'request_id': 'req-d80b4b46-ffb7-448c-a82b-08efa2517a7c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:44.960379 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] The token cache is being invalidate because project 2b5d5f73035444e3959e308137a08fe6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:54:44.961152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:44.962268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:44.968969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 843/1675] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:54:44 2022] DELETE /identity/v3/projects/2b5d5f73035444e3959e308137a08fe6 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:44.970635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:44.972804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:45.056707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 833/1676] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:44 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:45.060117 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0871d7e-a8e7-4449-b069-24bcd4021a30 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:45.064069 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f0871d7e-a8e7-4449-b069-24bcd4021a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:45.134261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:45.141355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80b4b46-ffb7-448c-a82b-08efa2517a7c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:45.155279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88377c41-fcc8-43d2-adc7-188a6754f55e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:45.156429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88377c41-fcc8-43d2-adc7-188a6754f55e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:45.157058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88377c41-fcc8-43d2-adc7-188a6754f55e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:45.157628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88377c41-fcc8-43d2-adc7-188a6754f55e admin admin] PATH_INFO: `/v3/projects/0010f98a56a04cbfbaafdfe771287dce/users/204c3f11e6e44d52b2d695850067f537/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:45.182642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88377c41-fcc8-43d2-adc7-188a6754f55e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0010f98a56a04cbfbaafdfe771287dce, user_id=204c3f11e6e44d52b2d695850067f537, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:45.194983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88377c41-fcc8-43d2-adc7-188a6754f55e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:45.211964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0871d7e-a8e7-4449-b069-24bcd4021a30 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:45.218223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0195eb4d-2843-4163-8afa-8488c2a52f0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:45.219162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0195eb4d-2843-4163-8afa-8488c2a52f0d service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:45.219545 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0195eb4d-2843-4163-8afa-8488c2a52f0d service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:45.219927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0195eb4d-2843-4163-8afa-8488c2a52f0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:45.226198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 844/1677] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:44 2022] PUT /identity/v3/projects/0010f98a56a04cbfbaafdfe771287dce/users/204c3f11e6e44d52b2d695850067f537/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:45.226198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7b4c8a-0cc6-4a55-aeab-5d337697cc22 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:45.226198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7b4c8a-0cc6-4a55-aeab-5d337697cc22 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:45.226198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7b4c8a-0cc6-4a55-aeab-5d337697cc22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:45.244753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0195eb4d-2843-4163-8afa-8488c2a52f0d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:45.260289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0195eb4d-2843-4163-8afa-8488c2a52f0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:45.261852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0195eb4d-2843-4163-8afa-8488c2a52f0d service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:45.272530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cc7b4c8a-0cc6-4a55-aeab-5d337697cc22 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:45.273774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cc7b4c8a-0cc6-4a55-aeab-5d337697cc22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:45.292399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 834/1678] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:45.293448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0195eb4d-2843-4163-8afa-8488c2a52f0d service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:45.294972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0195eb4d-2843-4163-8afa-8488c2a52f0d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:45.305028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:45.400739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 845/1679] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:45.404033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4dd57-3772-4e53-bdba-7d4a2af3ee0a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:45.404640 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4dd57-3772-4e53-bdba-7d4a2af3ee0a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:45.405037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e4dd57-3772-4e53-bdba-7d4a2af3ee0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:45.441255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0195eb4d-2843-4163-8afa-8488c2a52f0d service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:45.483339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-75e4dd57-3772-4e53-bdba-7d4a2af3ee0a None None] MFA Rules not processed for user `204c3f11e6e44d52b2d695850067f537`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:45.485104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-75e4dd57-3772-4e53-bdba-7d4a2af3ee0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:45.557075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f08fb9-a21a-4d67-ac1b-d8f78ce49228 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:45.557933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f08fb9-a21a-4d67-ac1b-d8f78ce49228 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:45.558403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f08fb9-a21a-4d67-ac1b-d8f78ce49228 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:45.559071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f08fb9-a21a-4d67-ac1b-d8f78ce49228 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:45.562183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-40f08fb9-a21a-4d67-ac1b-d8f78ce49228 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:45.641118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 846/1680] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:45 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:45.644915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23bc6f0-be6c-4cc4-bb5c-79d29c901c03 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:45.645216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23bc6f0-be6c-4cc4-bb5c-79d29c901c03 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:45.645484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23bc6f0-be6c-4cc4-bb5c-79d29c901c03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:45.672772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f08fb9-a21a-4d67-ac1b-d8f78ce49228 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:45.675387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f08fb9-a21a-4d67-ac1b-d8f78ce49228 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:45.689925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e23bc6f0-be6c-4cc4-bb5c-79d29c901c03 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:45.694170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e23bc6f0-be6c-4cc4-bb5c-79d29c901c03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:45.704475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:45.723299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 847/1681] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:45.724268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23bc6f0-be6c-4cc4-bb5c-79d29c901c03 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:45.733638 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 835/1682] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:45 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:45.734670 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f08fb9-a21a-4d67-ac1b-d8f78ce49228 service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:45.746299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23bc6f0-be6c-4cc4-bb5c-79d29c901c03 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:45.760100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1608cf48-7725-4e25-b959-f02fe04bc68c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:45.761462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1608cf48-7725-4e25-b959-f02fe04bc68c admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:45.761887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1608cf48-7725-4e25-b959-f02fe04bc68c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:45.762369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1608cf48-7725-4e25-b959-f02fe04bc68c admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:45.763069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1608cf48-7725-4e25-b959-f02fe04bc68c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:45.769946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1608cf48-7725-4e25-b959-f02fe04bc68c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:45.771313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f08fb9-a21a-4d67-ac1b-d8f78ce49228 service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:45.786213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eeacf7e-6d7c-4313-8b38-8c3f7042dfbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:45.787364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eeacf7e-6d7c-4313-8b38-8c3f7042dfbc admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:45.787935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eeacf7e-6d7c-4313-8b38-8c3f7042dfbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:45.787935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eeacf7e-6d7c-4313-8b38-8c3f7042dfbc admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:45.788555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eeacf7e-6d7c-4313-8b38-8c3f7042dfbc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:45.788858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eeacf7e-6d7c-4313-8b38-8c3f7042dfbc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:45.794221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eeacf7e-6d7c-4313-8b38-8c3f7042dfbc admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:45.834137 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 848/1683] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:45 2022] POST /identity/v3/projects => generated 369 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:45.834562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 836/1684] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:45.837568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1608cf48-7725-4e25-b959-f02fe04bc68c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:45.838493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eeacf7e-6d7c-4313-8b38-8c3f7042dfbc admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:45.877254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eeacf7e-6d7c-4313-8b38-8c3f7042dfbc admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:46.088657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1608cf48-7725-4e25-b959-f02fe04bc68c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:46.093592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a546cda4-eba4-40bb-b884-2faa8e735ce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:46.094265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a546cda4-eba4-40bb-b884-2faa8e735ce1 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.094539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a546cda4-eba4-40bb-b884-2faa8e735ce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.094804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a546cda4-eba4-40bb-b884-2faa8e735ce1 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.095440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a546cda4-eba4-40bb-b884-2faa8e735ce1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:46.099726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a546cda4-eba4-40bb-b884-2faa8e735ce1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:46.108512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566de508-56ee-4d42-9a52-1bd036cbd499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:46.109165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566de508-56ee-4d42-9a52-1bd036cbd499 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.109424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566de508-56ee-4d42-9a52-1bd036cbd499 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.109684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566de508-56ee-4d42-9a52-1bd036cbd499 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.110417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566de508-56ee-4d42-9a52-1bd036cbd499 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:46.115374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566de508-56ee-4d42-9a52-1bd036cbd499 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:46.149161 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 849/1685] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:45 2022] POST /identity/v3/users => generated 291 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:46.153060 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a546cda4-eba4-40bb-b884-2faa8e735ce1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:46.177603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a546cda4-eba4-40bb-b884-2faa8e735ce1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:46.192303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db05de5-0739-4a40-bc03-41a92e5adec8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:46.193877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db05de5-0739-4a40-bc03-41a92e5adec8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.195386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db05de5-0739-4a40-bc03-41a92e5adec8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.195817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db05de5-0739-4a40-bc03-41a92e5adec8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.197009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 837/1686] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:45 2022] POST /identity/v3/projects => generated 373 bytes in 359 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:46.197122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db05de5-0739-4a40-bc03-41a92e5adec8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:46.199101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db05de5-0739-4a40-bc03-41a92e5adec8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:46.200215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566de508-56ee-4d42-9a52-1bd036cbd499 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:46.206282 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:46.254704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 850/1687] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:46 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:46.255802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db05de5-0739-4a40-bc03-41a92e5adec8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:46.294898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db05de5-0739-4a40-bc03-41a92e5adec8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:46.403622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dc5531-ef6a-486e-8719-2e7ca693f1f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:46.404305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dc5531-ef6a-486e-8719-2e7ca693f1f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.404580 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dc5531-ef6a-486e-8719-2e7ca693f1f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.404847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dc5531-ef6a-486e-8719-2e7ca693f1f4 admin admin] PATH_INFO: `/v3/projects/1613eb1e67d342279ad00aa71293c088/users/8b72239238f54eef8fe1759067773736/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.416279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566de508-56ee-4d42-9a52-1bd036cbd499 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:46.425112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a98e318-a62a-4974-aa64-420b14c17fa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:46.426281 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a98e318-a62a-4974-aa64-420b14c17fa5 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.426703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dc5531-ef6a-486e-8719-2e7ca693f1f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1613eb1e67d342279ad00aa71293c088, user_id=8b72239238f54eef8fe1759067773736, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:46.426873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a98e318-a62a-4974-aa64-420b14c17fa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.427338 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a98e318-a62a-4974-aa64-420b14c17fa5 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.428136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a98e318-a62a-4974-aa64-420b14c17fa5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:46.429691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dc5531-ef6a-486e-8719-2e7ca693f1f4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:46.430633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a98e318-a62a-4974-aa64-420b14c17fa5 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:46.451853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 851/1688] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:46 2022] PUT /identity/v3/projects/1613eb1e67d342279ad00aa71293c088/users/8b72239238f54eef8fe1759067773736/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:46.451853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dc5531-ef6a-486e-8719-2e7ca693f1f4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:46.455449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c9dc5531-ef6a-486e-8719-2e7ca693f1f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:46.489639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 838/1689] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:46 2022] POST /identity/v3/users => generated 287 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:46.491160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a85c581-1468-4906-8629-89468c1f3bf6 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.491426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a85c581-1468-4906-8629-89468c1f3bf6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.491705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a85c581-1468-4906-8629-89468c1f3bf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.567535 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9a85c581-1468-4906-8629-89468c1f3bf6 None None] MFA Rules not processed for user `a8894fd1044241ee8ba84927b6e8326a`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:46.569222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9a85c581-1468-4906-8629-89468c1f3bf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:46.611904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dc5531-ef6a-486e-8719-2e7ca693f1f4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:46.618166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa60c3f-07a0-498d-b278-40553439bd3f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:46.618768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa60c3f-07a0-498d-b278-40553439bd3f service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.619038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa60c3f-07a0-498d-b278-40553439bd3f service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.619306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa60c3f-07a0-498d-b278-40553439bd3f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.621213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5fa60c3f-07a0-498d-b278-40553439bd3f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:46.649435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa60c3f-07a0-498d-b278-40553439bd3f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:46.654858 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa60c3f-07a0-498d-b278-40553439bd3f service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:46.678481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:46.686633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 852/1690] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:46.687537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa60c3f-07a0-498d-b278-40553439bd3f service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:46.706913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 839/1691] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:46 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:46.718390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2d9dc7-181a-4552-83ee-c34b0259ea62 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.718783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2d9dc7-181a-4552-83ee-c34b0259ea62 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.719183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2d9dc7-181a-4552-83ee-c34b0259ea62 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.721378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 840/1692] 10.222.0.25 () {54 vars in 912 bytes} [Sun Oct 2 00:54:46 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:54:46.722217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2d9dc7-181a-4552-83ee-c34b0259ea62 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:46.809781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa60c3f-07a0-498d-b278-40553439bd3f service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:46.823352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aba2b93-f48b-46f8-9799-48b8bb255f85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:46.825058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aba2b93-f48b-46f8-9799-48b8bb255f85 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.825338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aba2b93-f48b-46f8-9799-48b8bb255f85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.825608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aba2b93-f48b-46f8-9799-48b8bb255f85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.826281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aba2b93-f48b-46f8-9799-48b8bb255f85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:46.829384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aba2b93-f48b-46f8-9799-48b8bb255f85 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:46.853608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 853/1693] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:46 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:46.853608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aba2b93-f48b-46f8-9799-48b8bb255f85 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:46.867680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2d9dc7-181a-4552-83ee-c34b0259ea62 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:46.874543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8723bee6-75f7-4521-9101-ec71ab24c2bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:46.875251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8723bee6-75f7-4521-9101-ec71ab24c2bf admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.875754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8723bee6-75f7-4521-9101-ec71ab24c2bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.876099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8723bee6-75f7-4521-9101-ec71ab24c2bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.876821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8723bee6-75f7-4521-9101-ec71ab24c2bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:46.878682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8723bee6-75f7-4521-9101-ec71ab24c2bf admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:46.882209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9aba2b93-f48b-46f8-9799-48b8bb255f85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:46.901579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:46.912748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 841/1694] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:46 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:46.914364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d9951-21b0-4d96-8533-5168096640e2 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:46.916243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d9951-21b0-4d96-8533-5168096640e2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:46.916520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767d9951-21b0-4d96-8533-5168096640e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:46.945049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-767d9951-21b0-4d96-8533-5168096640e2 None None] MFA Rules not processed for user `e75811ee9ed7490985fed336ab9feb19`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:46.947266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-767d9951-21b0-4d96-8533-5168096640e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:47.059973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 842/1695] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 00:54:46 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:47.061391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767d9951-21b0-4d96-8533-5168096640e2 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:47.079582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aba2b93-f48b-46f8-9799-48b8bb255f85 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:47.086594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767d9951-21b0-4d96-8533-5168096640e2 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:47.093938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30030c54-15c6-4001-9a81-c67fff96879c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:47.094607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30030c54-15c6-4001-9a81-c67fff96879c service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:47.094866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30030c54-15c6-4001-9a81-c67fff96879c service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:47.095868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30030c54-15c6-4001-9a81-c67fff96879c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:47.100075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a27712-4c0b-418f-9b7e-1c2d04772ea1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:47.101195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a27712-4c0b-418f-9b7e-1c2d04772ea1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:47.101469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a27712-4c0b-418f-9b7e-1c2d04772ea1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:47.102384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a27712-4c0b-418f-9b7e-1c2d04772ea1 admin admin] PATH_INFO: `/v3/projects/1613eb1e67d342279ad00aa71293c088/users/8b72239238f54eef8fe1759067773736/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:47.107470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a27712-4c0b-418f-9b7e-1c2d04772ea1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1613eb1e67d342279ad00aa71293c088, user_id=8b72239238f54eef8fe1759067773736, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:47.112468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a27712-4c0b-418f-9b7e-1c2d04772ea1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:47.114973 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30030c54-15c6-4001-9a81-c67fff96879c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:47.118723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30030c54-15c6-4001-9a81-c67fff96879c service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:47.126755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 843/1696] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:47 2022] PUT /identity/v3/projects/1613eb1e67d342279ad00aa71293c088/users/8b72239238f54eef8fe1759067773736/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:47.127847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a27712-4c0b-418f-9b7e-1c2d04772ea1 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:47.133301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 854/1697] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:46 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:47.134267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30030c54-15c6-4001-9a81-c67fff96879c service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:47.159497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30030c54-15c6-4001-9a81-c67fff96879c service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:47.295395 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19763a48-4644-4a2f-be09-eff59ad9b0ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:47.296182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19763a48-4644-4a2f-be09-eff59ad9b0ac service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:47.296273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a27712-4c0b-418f-9b7e-1c2d04772ea1 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:47.296359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19763a48-4644-4a2f-be09-eff59ad9b0ac service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:47.296359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19763a48-4644-4a2f-be09-eff59ad9b0ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:47.313359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1519afd-1791-49c9-ac0e-85609737fdbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:47.315565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1519afd-1791-49c9-ac0e-85609737fdbd admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:47.315925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1519afd-1791-49c9-ac0e-85609737fdbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:47.317026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1519afd-1791-49c9-ac0e-85609737fdbd admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:47.336651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1519afd-1791-49c9-ac0e-85609737fdbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:47.342309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19763a48-4644-4a2f-be09-eff59ad9b0ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:47.343923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19763a48-4644-4a2f-be09-eff59ad9b0ac service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:47.344887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1519afd-1791-49c9-ac0e-85609737fdbd admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:47.381918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 844/1698] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:47 2022] GET /identity/v3/roles => generated 1420 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:47.385299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 855/1699] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:47 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:47.386761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab758fdb-f17e-4e04-86a0-b3b521b37c22 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:47.387970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1519afd-1791-49c9-ac0e-85609737fdbd admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:47.388503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab758fdb-f17e-4e04-86a0-b3b521b37c22 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:47.388799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab758fdb-f17e-4e04-86a0-b3b521b37c22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:47.410838 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1519afd-1791-49c9-ac0e-85609737fdbd admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:47.421526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c440ce6d-9b53-49ca-93c0-4861b31f3ae1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:47.422355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c440ce6d-9b53-49ca-93c0-4861b31f3ae1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:47.422524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c440ce6d-9b53-49ca-93c0-4861b31f3ae1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:47.423541 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c440ce6d-9b53-49ca-93c0-4861b31f3ae1 admin admin] PATH_INFO: `/v3/projects/81e2ad2846234ee999dd454bdea3299f/users/b71c040b0fcc4a8a91379722f7e79e4a/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:47.464092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ab758fdb-f17e-4e04-86a0-b3b521b37c22 None None] MFA Rules not processed for user `8b72239238f54eef8fe1759067773736`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:47.468030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c440ce6d-9b53-49ca-93c0-4861b31f3ae1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=81e2ad2846234ee999dd454bdea3299f, user_id=b71c040b0fcc4a8a91379722f7e79e4a, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:47.468958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ab758fdb-f17e-4e04-86a0-b3b521b37c22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:47.474192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c440ce6d-9b53-49ca-93c0-4861b31f3ae1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:47.484618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:47.527183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 845/1700] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:47 2022] PUT /identity/v3/projects/81e2ad2846234ee999dd454bdea3299f/users/b71c040b0fcc4a8a91379722f7e79e4a/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:47.528762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1511dde6-edd1-402c-821e-6a6be879b2c8 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:47.529074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1511dde6-edd1-402c-821e-6a6be879b2c8 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:47.529562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1511dde6-edd1-402c-821e-6a6be879b2c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:47.627527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-1511dde6-edd1-402c-821e-6a6be879b2c8 None None] MFA Rules not processed for user `b71c040b0fcc4a8a91379722f7e79e4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:47.629310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1511dde6-edd1-402c-821e-6a6be879b2c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:47.702398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 856/1701] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:47 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:47.715063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab758fdb-f17e-4e04-86a0-b3b521b37c22 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:47.716900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ab758fdb-f17e-4e04-86a0-b3b521b37c22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:47.773573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:47.824149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 846/1702] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:47 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:47.895433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab758fdb-f17e-4e04-86a0-b3b521b37c22 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:48.038890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1511dde6-edd1-402c-821e-6a6be879b2c8 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:48.047622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739003c0-0f0b-4437-9440-9b165b687268 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:48.048260 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739003c0-0f0b-4437-9440-9b165b687268 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:48.048542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739003c0-0f0b-4437-9440-9b165b687268 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:48.048816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739003c0-0f0b-4437-9440-9b165b687268 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:48.197917 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1511dde6-edd1-402c-821e-6a6be879b2c8 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:48.214398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1694444-6d22-4494-9a50-06121313298d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:48.215030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1694444-6d22-4494-9a50-06121313298d service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:48.215302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1694444-6d22-4494-9a50-06121313298d service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:48.215721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1694444-6d22-4494-9a50-06121313298d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:48.230987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1694444-6d22-4494-9a50-06121313298d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:48.233059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1694444-6d22-4494-9a50-06121313298d service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:48.236525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739003c0-0f0b-4437-9440-9b165b687268 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:48.240361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739003c0-0f0b-4437-9440-9b165b687268 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:48.264909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 847/1703] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:48 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:48.283265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 857/1704] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:47 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 569 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:54:48.764563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8487093c-29b8-4265-af71-46cf2ecbd51d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:48.764894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8487093c-29b8-4265-af71-46cf2ecbd51d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:48.765171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8487093c-29b8-4265-af71-46cf2ecbd51d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:48.794849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739003c0-0f0b-4437-9440-9b165b687268 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:48.812657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8487093c-29b8-4265-af71-46cf2ecbd51d None None] MFA Rules not processed for user `afe98894d83f4678bb134782d90706e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:48.820003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8487093c-29b8-4265-af71-46cf2ecbd51d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:48.948773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739003c0-0f0b-4437-9440-9b165b687268 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:48.955713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:48.957001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:48.957765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:48.959630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] PATH_INFO: `/v3/users/b71c040b0fcc4a8a91379722f7e79e4a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:48.961934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b71c040b0fcc4a8a91379722f7e79e4a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:48.965530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:49.005916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 848/1705] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:48 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:49.058875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:49.099928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b71c040b0fcc4a8a91379722f7e79e4a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:49.115067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b71c040b0fcc4a8a91379722f7e79e4a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:49.116760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b71c040b0fcc4a8a91379722f7e79e4a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:49.120057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b71c040b0fcc4a8a91379722f7e79e4a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:49.134960 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b71c040b0fcc4a8a91379722f7e79e4a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:49.176799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 858/1706] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:54:48 2022] DELETE /identity/v3/users/b71c040b0fcc4a8a91379722f7e79e4a => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:49.181243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed958c5b-5f54-4142-a9a1-68b83fbec07f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:49.181649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed958c5b-5f54-4142-a9a1-68b83fbec07f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:49.182019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed958c5b-5f54-4142-a9a1-68b83fbec07f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:49.261906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ed958c5b-5f54-4142-a9a1-68b83fbec07f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:49.273755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ed958c5b-5f54-4142-a9a1-68b83fbec07f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:49.305745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:49.468604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 849/1707] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:49.481818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:49.485389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:49.698427 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f579322-943d-458b-9899-69df126dc385 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:49.710127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af7d0ee-0b35-4dae-ab73-27f1e3185447 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:49.714908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af7d0ee-0b35-4dae-ab73-27f1e3185447 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:49.718132 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af7d0ee-0b35-4dae-ab73-27f1e3185447 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:49.718527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af7d0ee-0b35-4dae-ab73-27f1e3185447 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:49.750477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af7d0ee-0b35-4dae-ab73-27f1e3185447 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:49.752066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af7d0ee-0b35-4dae-ab73-27f1e3185447 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:49.803879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 859/1708] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:50.016856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed958c5b-5f54-4142-a9a1-68b83fbec07f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:50.018406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ed958c5b-5f54-4142-a9a1-68b83fbec07f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:50.053248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed958c5b-5f54-4142-a9a1-68b83fbec07f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:50.071371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:50.072249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:50.072673 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:50.074155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] PATH_INFO: `/v3/users/a852bc2fda7a479e879b7803d24530a0` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:50.079148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a852bc2fda7a479e879b7803d24530a0)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:50.088061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:50.198517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a852bc2fda7a479e879b7803d24530a0'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:50.201428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a852bc2fda7a479e879b7803d24530a0'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:50.223839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a852bc2fda7a479e879b7803d24530a0'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:50.228794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a852bc2fda7a479e879b7803d24530a0'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:50.248443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a852bc2fda7a479e879b7803d24530a0'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:50.303218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 850/1709] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:54:50 2022] DELETE /identity/v3/users/a852bc2fda7a479e879b7803d24530a0 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:50.311310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933e1e3-6154-43f7-a708-b9b28b64b383 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:50.311310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933e1e3-6154-43f7-a708-b9b28b64b383 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:50.311310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933e1e3-6154-43f7-a708-b9b28b64b383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:50.345211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4933e1e3-6154-43f7-a708-b9b28b64b383 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:50.346784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4933e1e3-6154-43f7-a708-b9b28b64b383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:50.489272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 860/1710] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:54:50.560231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:50.562089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:50.754778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89e807b-a27b-4233-bb35-4836e8e72472 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:50.769410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dc3713-1015-4a8b-a722-5d8ad1dd6b1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:50.771463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dc3713-1015-4a8b-a722-5d8ad1dd6b1c service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:50.772393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dc3713-1015-4a8b-a722-5d8ad1dd6b1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:50.773871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6dc3713-1015-4a8b-a722-5d8ad1dd6b1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:50.820136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dc3713-1015-4a8b-a722-5d8ad1dd6b1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:50.825649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6dc3713-1015-4a8b-a722-5d8ad1dd6b1c service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:50.861382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 851/1711] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:51.055826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a8cdfc-d2cd-4426-a479-a7141b081cc0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:51.057605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a8cdfc-d2cd-4426-a479-a7141b081cc0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:51.058198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a8cdfc-d2cd-4426-a479-a7141b081cc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:51.173210 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dc3713-1015-4a8b-a722-5d8ad1dd6b1c service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:51.174822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c6dc3713-1015-4a8b-a722-5d8ad1dd6b1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:51.227732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6dc3713-1015-4a8b-a722-5d8ad1dd6b1c service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:51.268272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:51.273226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:51.273542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-52a8cdfc-d2cd-4426-a479-a7141b081cc0 None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:51.273683 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:51.278850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] PATH_INFO: `/v3/projects/81e2ad2846234ee999dd454bdea3299f` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:51.282672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-52a8cdfc-d2cd-4426-a479-a7141b081cc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:51.316023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] RBAC: Authorizing `identity:delete_project(project_id=81e2ad2846234ee999dd454bdea3299f)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:51.328959 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:51.381895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:51.467354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:51.546540 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81e2ad2846234ee999dd454bdea3299f', 'request_id': 'req-c9a20b08-fec9-481f-a1c9-2c86193c143a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:51.566767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81e2ad2846234ee999dd454bdea3299f', 'request_id': 'req-c9a20b08-fec9-481f-a1c9-2c86193c143a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:51.576682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 861/1712] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 00:54:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 523 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:54:51.593134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81e2ad2846234ee999dd454bdea3299f', 'request_id': 'req-c9a20b08-fec9-481f-a1c9-2c86193c143a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:51.652986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] The token cache is being invalidate because project 81e2ad2846234ee999dd454bdea3299f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:54:51.653562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:51.658657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:51.660872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 852/1713] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:54:51 2022] DELETE /identity/v3/projects/81e2ad2846234ee999dd454bdea3299f => generated 0 bytes in 488 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:51.666637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f63a39-9972-4c37-ac52-92fd31c5f1c4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:51.666637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f63a39-9972-4c37-ac52-92fd31c5f1c4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:51.667125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f63a39-9972-4c37-ac52-92fd31c5f1c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:51.717475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-51f63a39-9972-4c37-ac52-92fd31c5f1c4 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:51.721355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-51f63a39-9972-4c37-ac52-92fd31c5f1c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:51.885124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:51.886875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:51.901145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 862/1714] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:54:51.910276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fd7499-96d1-46d3-b199-7b152b20738d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:51.910681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fd7499-96d1-46d3-b199-7b152b20738d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:51.910823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fd7499-96d1-46d3-b199-7b152b20738d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:52.003756 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-06fd7499-96d1-46d3-b199-7b152b20738d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:52.008717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-06fd7499-96d1-46d3-b199-7b152b20738d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:52.044483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 863/1715] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:52.050474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fd7499-96d1-46d3-b199-7b152b20738d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:52.097787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fd7499-96d1-46d3-b199-7b152b20738d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:52.108989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a66883-e034-4488-928b-d0de0a76df20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:52.109611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a66883-e034-4488-928b-d0de0a76df20 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:52.109941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a66883-e034-4488-928b-d0de0a76df20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:52.110527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a66883-e034-4488-928b-d0de0a76df20 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:52.111337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a66883-e034-4488-928b-d0de0a76df20 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:52.111830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a66883-e034-4488-928b-d0de0a76df20 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:52.113712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a66883-e034-4488-928b-d0de0a76df20 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:52.143732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 864/1716] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:52.148649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a66883-e034-4488-928b-d0de0a76df20 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:52.176736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a66883-e034-4488-928b-d0de0a76df20 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:52.189971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abc1da5-2d22-44a2-b618-1f38d6c9a473 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:52.190647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abc1da5-2d22-44a2-b618-1f38d6c9a473 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:52.190928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abc1da5-2d22-44a2-b618-1f38d6c9a473 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:52.191197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abc1da5-2d22-44a2-b618-1f38d6c9a473 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:52.191830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abc1da5-2d22-44a2-b618-1f38d6c9a473 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:52.206391 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abc1da5-2d22-44a2-b618-1f38d6c9a473 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:52.263147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a20b08-fec9-481f-a1c9-2c86193c143a admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:52.271988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026ac3f-b2d0-4727-9fdd-48d6b736a27a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:52.273046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026ac3f-b2d0-4727-9fdd-48d6b736a27a service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:52.274129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026ac3f-b2d0-4727-9fdd-48d6b736a27a service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:52.274580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026ac3f-b2d0-4727-9fdd-48d6b736a27a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:52.276816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1026ac3f-b2d0-4727-9fdd-48d6b736a27a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:52.286657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 865/1717] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:52 2022] POST /identity/v3/projects => generated 373 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:52.290531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abc1da5-2d22-44a2-b618-1f38d6c9a473 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:52.484082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1026ac3f-b2d0-4727-9fdd-48d6b736a27a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:52.487864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1026ac3f-b2d0-4727-9fdd-48d6b736a27a service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:52.516361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 853/1718] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 00:54:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 631 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 00:54:52.559385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abc1da5-2d22-44a2-b618-1f38d6c9a473 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:52.577711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ed7e7e-6710-48a0-bff1-a8142507a92c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:52.579361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ed7e7e-6710-48a0-bff1-a8142507a92c admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:52.582045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ed7e7e-6710-48a0-bff1-a8142507a92c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:52.584199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ed7e7e-6710-48a0-bff1-a8142507a92c admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:52.586354 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ed7e7e-6710-48a0-bff1-a8142507a92c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:52.592059 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ed7e7e-6710-48a0-bff1-a8142507a92c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:52.712903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 866/1719] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:52 2022] POST /identity/v3/users => generated 287 bytes in 423 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:54:52.717820 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026ac3f-b2d0-4727-9fdd-48d6b736a27a service swift] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:52.739063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026ac3f-b2d0-4727-9fdd-48d6b736a27a service swift] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:52.786104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b59b2a-9924-4f42-9bff-2a5dbd239270 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:52.788887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b59b2a-9924-4f42-9bff-2a5dbd239270 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:52.791101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b59b2a-9924-4f42-9bff-2a5dbd239270 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:52.791591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b59b2a-9924-4f42-9bff-2a5dbd239270 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:52.795662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b59b2a-9924-4f42-9bff-2a5dbd239270 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:52.799515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b59b2a-9924-4f42-9bff-2a5dbd239270 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:52.879078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ed7e7e-6710-48a0-bff1-a8142507a92c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:52.879602 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 854/1720] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:52 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:52.916595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ed7e7e-6710-48a0-bff1-a8142507a92c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:52.933744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c7b536-8e8d-4314-a222-b95598c10915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:52.936874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c7b536-8e8d-4314-a222-b95598c10915 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:52.939246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c7b536-8e8d-4314-a222-b95598c10915 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:52.940746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c7b536-8e8d-4314-a222-b95598c10915 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:52.946730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c7b536-8e8d-4314-a222-b95598c10915 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:52.966545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c7b536-8e8d-4314-a222-b95598c10915 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:52.968196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:53.000342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 867/1721] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:52 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:53.004783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b59b2a-9924-4f42-9bff-2a5dbd239270 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:53.020677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b59b2a-9924-4f42-9bff-2a5dbd239270 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:53.051483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a63c5b5-0ffe-4658-8058-2ec3e0501fe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:53.052255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a63c5b5-0ffe-4658-8058-2ec3e0501fe3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:53.066446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a63c5b5-0ffe-4658-8058-2ec3e0501fe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:53.067035 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a63c5b5-0ffe-4658-8058-2ec3e0501fe3 admin admin] PATH_INFO: `/v3/projects/21effdef168e4e4ab8e5fffcdb4c9c95/users/eb09b2d131cf45ef8ae8ade4b101f710/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:53.172685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a63c5b5-0ffe-4658-8058-2ec3e0501fe3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21effdef168e4e4ab8e5fffcdb4c9c95, user_id=eb09b2d131cf45ef8ae8ade4b101f710, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:53.187672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a63c5b5-0ffe-4658-8058-2ec3e0501fe3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:53.217929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:53.251180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fd958d-f3f2-4a23-8858-a3cf8e576bb0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:53.252071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fd958d-f3f2-4a23-8858-a3cf8e576bb0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:53.252318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 855/1722] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:53 2022] PUT /identity/v3/projects/21effdef168e4e4ab8e5fffcdb4c9c95/users/eb09b2d131cf45ef8ae8ade4b101f710/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:53.252784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fd958d-f3f2-4a23-8858-a3cf8e576bb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:53.368939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-85fd958d-f3f2-4a23-8858-a3cf8e576bb0 None None] MFA Rules not processed for user `eb09b2d131cf45ef8ae8ade4b101f710`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:53.375819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-85fd958d-f3f2-4a23-8858-a3cf8e576bb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:53.574520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a63c5b5-0ffe-4658-8058-2ec3e0501fe3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:53.577847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6a63c5b5-0ffe-4658-8058-2ec3e0501fe3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:53.670861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 868/1723] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 421 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:54:53.680063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fd958d-f3f2-4a23-8858-a3cf8e576bb0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:53.888392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a63c5b5-0ffe-4658-8058-2ec3e0501fe3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:53.906700 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:53.910645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:53.912379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:53.912875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] PATH_INFO: `/v3/users/204c3f11e6e44d52b2d695850067f537` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:53.919067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] RBAC: Authorizing `identity:delete_user(user_id=204c3f11e6e44d52b2d695850067f537)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:53.924093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:54.032670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fd958d-f3f2-4a23-8858-a3cf8e576bb0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:54.063404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02cfc83-a8fc-4a0f-be11-31d3f3d14700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:54.064076 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02cfc83-a8fc-4a0f-be11-31d3f3d14700 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:54.067212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02cfc83-a8fc-4a0f-be11-31d3f3d14700 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:54.069345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02cfc83-a8fc-4a0f-be11-31d3f3d14700 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:54.071715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02cfc83-a8fc-4a0f-be11-31d3f3d14700 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:54.077712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e02cfc83-a8fc-4a0f-be11-31d3f3d14700 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:54.083159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '204c3f11e6e44d52b2d695850067f537'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.088179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '204c3f11e6e44d52b2d695850067f537'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.145078 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '204c3f11e6e44d52b2d695850067f537'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.150328 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '204c3f11e6e44d52b2d695850067f537'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.180862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '204c3f11e6e44d52b2d695850067f537'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.211360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 869/1724] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:53 2022] POST /identity/v3/projects => generated 373 bytes in 532 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:54.214796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02cfc83-a8fc-4a0f-be11-31d3f3d14700 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:54.223712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e02cfc83-a8fc-4a0f-be11-31d3f3d14700 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:54.307437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 856/1725] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:54:53 2022] DELETE /identity/v3/users/204c3f11e6e44d52b2d695850067f537 => generated 0 bytes in 734 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:54.325207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:54.329350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:54.457274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02cfc83-a8fc-4a0f-be11-31d3f3d14700 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:54.518107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:54.571742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8472f-1e9f-44cf-a1c3-18dbaf27541b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:54.591787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:54.592558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:54.592779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:54.593030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] PATH_INFO: `/v3/users/7932a6409fba4ccaaddbc6e5cb93709b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:54.598571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7932a6409fba4ccaaddbc6e5cb93709b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:54.605712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:54.675643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:54.677341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:54.678040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:54.678767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] PATH_INFO: `/v3/projects/9a3b40887e3b44478be649dfaa3f0322` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:54.684667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a3b40887e3b44478be649dfaa3f0322)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:54.687526 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:54.689821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:54.753743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7932a6409fba4ccaaddbc6e5cb93709b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.755825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7932a6409fba4ccaaddbc6e5cb93709b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.788531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7932a6409fba4ccaaddbc6e5cb93709b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.792550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7932a6409fba4ccaaddbc6e5cb93709b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.830105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7932a6409fba4ccaaddbc6e5cb93709b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.847695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a3b40887e3b44478be649dfaa3f0322', 'request_id': 'req-fb17d4a4-aba4-4885-820a-6c86156a5083'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.864835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a3b40887e3b44478be649dfaa3f0322', 'request_id': 'req-fb17d4a4-aba4-4885-820a-6c86156a5083'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.899648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a3b40887e3b44478be649dfaa3f0322', 'request_id': 'req-fb17d4a4-aba4-4885-820a-6c86156a5083'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.921055 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 857/1726] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:54:54 2022] DELETE /identity/v3/users/7932a6409fba4ccaaddbc6e5cb93709b => generated 0 bytes in 602 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:54.921725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:54.925192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:54.956016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] The token cache is being invalidate because project 9a3b40887e3b44478be649dfaa3f0322 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:54:54.958991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.961081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fb17d4a4-aba4-4885-820a-6c86156a5083 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:54.972345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 870/1727] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:54:54 2022] DELETE /identity/v3/projects/9a3b40887e3b44478be649dfaa3f0322 => generated 0 bytes in 754 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:54.976724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d47b6c3-9e9e-4d95-9f0b-7dbcb5181990 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:54.976724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d47b6c3-9e9e-4d95-9f0b-7dbcb5181990 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:54.977054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d47b6c3-9e9e-4d95-9f0b-7dbcb5181990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:55.069958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1d47b6c3-9e9e-4d95-9f0b-7dbcb5181990 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:55.071637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1d47b6c3-9e9e-4d95-9f0b-7dbcb5181990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:55.255589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe024a1-9344-494f-822a-4a16b45bee31 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:55.264225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482c6002-8135-41d5-b58d-5d74ba1f0582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:55.270665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482c6002-8135-41d5-b58d-5d74ba1f0582 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:55.270665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482c6002-8135-41d5-b58d-5d74ba1f0582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:55.270665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482c6002-8135-41d5-b58d-5d74ba1f0582 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:55.270665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482c6002-8135-41d5-b58d-5d74ba1f0582 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:55.272153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482c6002-8135-41d5-b58d-5d74ba1f0582 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:55.306391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 858/1728] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:54 2022] POST /identity/v3/users => generated 293 bytes in 383 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:55.308504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641dd76-0f91-4dd8-9e7e-4f46a16873c6 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:55.309369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641dd76-0f91-4dd8-9e7e-4f46a16873c6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:55.310040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641dd76-0f91-4dd8-9e7e-4f46a16873c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:55.340878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 871/1729] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:55.342350 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aafe8fa-45b1-4a69-b93d-d1878c9fa7de None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:55.342636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aafe8fa-45b1-4a69-b93d-d1878c9fa7de None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:55.342910 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aafe8fa-45b1-4a69-b93d-d1878c9fa7de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:55.402223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8aafe8fa-45b1-4a69-b93d-d1878c9fa7de None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:55.403549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8aafe8fa-45b1-4a69-b93d-d1878c9fa7de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:55.410854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-7641dd76-0f91-4dd8-9e7e-4f46a16873c6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:55.413918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7641dd76-0f91-4dd8-9e7e-4f46a16873c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:55.440623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 872/1730] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:55.444342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be24548-ef2d-47b4-9f60-e028a836d2dc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:55.449255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be24548-ef2d-47b4-9f60-e028a836d2dc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:55.449902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be24548-ef2d-47b4-9f60-e028a836d2dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:55.480595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 859/1731] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:55.481624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7641dd76-0f91-4dd8-9e7e-4f46a16873c6 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:55.488350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7641dd76-0f91-4dd8-9e7e-4f46a16873c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:55.518124 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3be24548-ef2d-47b4-9f60-e028a836d2dc None None] MFA Rules not processed for user `a8894fd1044241ee8ba84927b6e8326a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:55.523258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3be24548-ef2d-47b4-9f60-e028a836d2dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:55.548452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7641dd76-0f91-4dd8-9e7e-4f46a16873c6 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:55.562925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca5b677-e867-419c-8fb0-1d6b590b40c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:55.564099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca5b677-e867-419c-8fb0-1d6b590b40c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:55.564734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca5b677-e867-419c-8fb0-1d6b590b40c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:55.565263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca5b677-e867-419c-8fb0-1d6b590b40c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:55.566031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca5b677-e867-419c-8fb0-1d6b590b40c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:55.570699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca5b677-e867-419c-8fb0-1d6b590b40c2 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:55.605364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 860/1732] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:55 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:55.607377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca5b677-e867-419c-8fb0-1d6b590b40c2 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:55.611915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:55.617616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6ca5b677-e867-419c-8fb0-1d6b590b40c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:55.686634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 873/1733] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:55 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:55.696817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c7f5e5-9b29-4a4d-a78c-9d5e7a1ca55b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:55.700925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c7f5e5-9b29-4a4d-a78c-9d5e7a1ca55b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:55.701410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c7f5e5-9b29-4a4d-a78c-9d5e7a1ca55b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:55.784642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b1c7f5e5-9b29-4a4d-a78c-9d5e7a1ca55b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:55.790021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b1c7f5e5-9b29-4a4d-a78c-9d5e7a1ca55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:55.816108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 874/1734] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:55.818524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c7f5e5-9b29-4a4d-a78c-9d5e7a1ca55b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:55.826995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca5b677-e867-419c-8fb0-1d6b590b40c2 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:55.833537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e26866-9b58-4b8e-95bf-73091d67e983 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:55.834612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e26866-9b58-4b8e-95bf-73091d67e983 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:55.835139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e26866-9b58-4b8e-95bf-73091d67e983 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:55.835516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e26866-9b58-4b8e-95bf-73091d67e983 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:55.844269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c7f5e5-9b29-4a4d-a78c-9d5e7a1ca55b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:55.859093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e31d617-7041-411c-ac7d-515f123bbda6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:55.859950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e31d617-7041-411c-ac7d-515f123bbda6 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:55.860373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e31d617-7041-411c-ac7d-515f123bbda6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:55.860789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e31d617-7041-411c-ac7d-515f123bbda6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:55.864979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e26866-9b58-4b8e-95bf-73091d67e983 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:55.868093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e26866-9b58-4b8e-95bf-73091d67e983 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:55.888341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 861/1735] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:55.890909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e26866-9b58-4b8e-95bf-73091d67e983 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:55.898131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e31d617-7041-411c-ac7d-515f123bbda6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:55.899649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e31d617-7041-411c-ac7d-515f123bbda6 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:55.903972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e26866-9b58-4b8e-95bf-73091d67e983 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:55.909920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4411bc55-1dec-4b84-bcb6-1f28cc9ed04d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:55.910723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411bc55-1dec-4b84-bcb6-1f28cc9ed04d admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:55.911106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411bc55-1dec-4b84-bcb6-1f28cc9ed04d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:55.912067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4411bc55-1dec-4b84-bcb6-1f28cc9ed04d admin admin] PATH_INFO: `/v3/projects/5d513486accb4b71a08a5c2d8d79ea55/users/0b5649b0230449288817788f3d4f874f/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:55.948898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 875/1736] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:55.949933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e31d617-7041-411c-ac7d-515f123bbda6 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:55.953445 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:55.977479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4411bc55-1dec-4b84-bcb6-1f28cc9ed04d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d513486accb4b71a08a5c2d8d79ea55, user_id=0b5649b0230449288817788f3d4f874f, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:55.985503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4411bc55-1dec-4b84-bcb6-1f28cc9ed04d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:56.011915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e31d617-7041-411c-ac7d-515f123bbda6 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:56.018650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 862/1737] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:55 2022] PUT /identity/v3/projects/5d513486accb4b71a08a5c2d8d79ea55/users/0b5649b0230449288817788f3d4f874f/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:56.020791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd34e9a-833b-4997-b500-77a7df23f940 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:56.023656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd34e9a-833b-4997-b500-77a7df23f940 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:56.024159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd34e9a-833b-4997-b500-77a7df23f940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:56.087229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4bd34e9a-833b-4997-b500-77a7df23f940 None None] MFA Rules not processed for user `e75811ee9ed7490985fed336ab9feb19`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:56.103650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4bd34e9a-833b-4997-b500-77a7df23f940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:56.302500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afa9f60-fd11-4fd5-bdfd-f937bf445850 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:56.303924 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afa9f60-fd11-4fd5-bdfd-f937bf445850 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:56.303924 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afa9f60-fd11-4fd5-bdfd-f937bf445850 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:56.305846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afa9f60-fd11-4fd5-bdfd-f937bf445850 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:56.307665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afa9f60-fd11-4fd5-bdfd-f937bf445850 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:54:56.308917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afa9f60-fd11-4fd5-bdfd-f937bf445850 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:56.312784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afa9f60-fd11-4fd5-bdfd-f937bf445850 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:56.323206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 863/1738] 10.222.0.43 () {58 vars in 1015 bytes} [Sun Oct 2 00:54:56 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:56.324394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd34e9a-833b-4997-b500-77a7df23f940 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:56.354036 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 876/1739] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:54:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 404 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:54:56.357768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afa9f60-fd11-4fd5-bdfd-f937bf445850 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:56.360371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd34e9a-833b-4997-b500-77a7df23f940 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:56.373410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9069a75-e313-4f26-b389-a3171ed9e1f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:56.374661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9069a75-e313-4f26-b389-a3171ed9e1f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:56.375079 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9069a75-e313-4f26-b389-a3171ed9e1f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:56.376588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9069a75-e313-4f26-b389-a3171ed9e1f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:56.378736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9069a75-e313-4f26-b389-a3171ed9e1f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:56.384591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afa9f60-fd11-4fd5-bdfd-f937bf445850 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:56.385499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9069a75-e313-4f26-b389-a3171ed9e1f9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:56.396875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c6af37-457a-47a6-ad60-7b2c0af87b2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:56.397939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6af37-457a-47a6-ad60-7b2c0af87b2a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:56.399189 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6af37-457a-47a6-ad60-7b2c0af87b2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:56.399189 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c6af37-457a-47a6-ad60-7b2c0af87b2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:56.401293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c6af37-457a-47a6-ad60-7b2c0af87b2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:56.405220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c6af37-457a-47a6-ad60-7b2c0af87b2a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:56.412632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 864/1740] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:56 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:56.413746 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9069a75-e313-4f26-b389-a3171ed9e1f9 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:56.420744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e9069a75-e313-4f26-b389-a3171ed9e1f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:56.451819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9069a75-e313-4f26-b389-a3171ed9e1f9 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:56.460688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2c7043-27df-4f81-91d7-30babbf12aa1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:56.466686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2c7043-27df-4f81-91d7-30babbf12aa1 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:56.467093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2c7043-27df-4f81-91d7-30babbf12aa1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:56.467941 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2c7043-27df-4f81-91d7-30babbf12aa1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:56.473715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 877/1741] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:54:56 2022] POST /identity/v3/projects => generated 403 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:56.476345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c6af37-457a-47a6-ad60-7b2c0af87b2a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:56.492891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2c7043-27df-4f81-91d7-30babbf12aa1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:56.496000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2c7043-27df-4f81-91d7-30babbf12aa1 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:56.516937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 865/1742] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:56.517895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2c7043-27df-4f81-91d7-30babbf12aa1 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:56.538557 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2c7043-27df-4f81-91d7-30babbf12aa1 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:56.608226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c6af37-457a-47a6-ad60-7b2c0af87b2a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:56.627693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271b862d-a06e-44ba-93f2-05d3db047d56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:56.629753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b862d-a06e-44ba-93f2-05d3db047d56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:56.631933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b862d-a06e-44ba-93f2-05d3db047d56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:56.632812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b862d-a06e-44ba-93f2-05d3db047d56 admin admin] PATH_INFO: `/v3/projects/5d513486accb4b71a08a5c2d8d79ea55/users/0b5649b0230449288817788f3d4f874f/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:56.638984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271b862d-a06e-44ba-93f2-05d3db047d56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d513486accb4b71a08a5c2d8d79ea55, user_id=0b5649b0230449288817788f3d4f874f, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:56.648218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271b862d-a06e-44ba-93f2-05d3db047d56 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:56.690522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 878/1743] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:56 2022] PUT /identity/v3/projects/5d513486accb4b71a08a5c2d8d79ea55/users/0b5649b0230449288817788f3d4f874f/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:56.701614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35448401-ca53-45cc-b891-99d44dd286e0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:56.708921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35448401-ca53-45cc-b891-99d44dd286e0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:56.709902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35448401-ca53-45cc-b891-99d44dd286e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:56.715928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfe025e-8ace-4255-8bfd-256dafcbe22a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:56.717472 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfe025e-8ace-4255-8bfd-256dafcbe22a admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:56.720723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfe025e-8ace-4255-8bfd-256dafcbe22a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:56.721227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfe025e-8ace-4255-8bfd-256dafcbe22a admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:56.725149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfe025e-8ace-4255-8bfd-256dafcbe22a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:56.732207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfe025e-8ace-4255-8bfd-256dafcbe22a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:56.811140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 866/1744] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:54:56 2022] POST /identity/v3/users => generated 302 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:54:56.815532 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfe025e-8ace-4255-8bfd-256dafcbe22a admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:56.827244 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-35448401-ca53-45cc-b891-99d44dd286e0 None None] MFA Rules not processed for user `0b5649b0230449288817788f3d4f874f`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:56.837845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-35448401-ca53-45cc-b891-99d44dd286e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:56.862277 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfe025e-8ace-4255-8bfd-256dafcbe22a admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:56.927157 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:57.063664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7020f3-c8ca-43db-9189-3186eaecbd85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:57.065409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7020f3-c8ca-43db-9189-3186eaecbd85 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:57.065409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7020f3-c8ca-43db-9189-3186eaecbd85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:57.067030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7020f3-c8ca-43db-9189-3186eaecbd85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:57.067770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7020f3-c8ca-43db-9189-3186eaecbd85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:57.069400 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7020f3-c8ca-43db-9189-3186eaecbd85 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:57.076575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 879/1745] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:56 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 376 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:57.089691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:57.091297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35448401-ca53-45cc-b891-99d44dd286e0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:57.100871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 867/1746] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:56 2022] GET /identity/v3/roles => generated 1420 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:57.102013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7020f3-c8ca-43db-9189-3186eaecbd85 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:57.107698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35448401-ca53-45cc-b891-99d44dd286e0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:57.110555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0d7020f3-c8ca-43db-9189-3186eaecbd85 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:57.118413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482115cf-4023-48f3-b3b9-22ae234f2915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:57.119801 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482115cf-4023-48f3-b3b9-22ae234f2915 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:57.120235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482115cf-4023-48f3-b3b9-22ae234f2915 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:57.124236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482115cf-4023-48f3-b3b9-22ae234f2915 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:57.124236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482115cf-4023-48f3-b3b9-22ae234f2915 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:57.127793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482115cf-4023-48f3-b3b9-22ae234f2915 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:57.155820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 880/1747] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:54:57 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:54:57.159765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482115cf-4023-48f3-b3b9-22ae234f2915 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:57.190479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482115cf-4023-48f3-b3b9-22ae234f2915 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:57.210976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70036939-48c9-4087-9de9-40d6d332c6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:57.210976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70036939-48c9-4087-9de9-40d6d332c6b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:57.210976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70036939-48c9-4087-9de9-40d6d332c6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:57.210976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70036939-48c9-4087-9de9-40d6d332c6b0 admin admin] PATH_INFO: `/v3/projects/1910803c2a18420489568bb3b2bebe64/users/421390792a8346a2a5ae094ee4549582/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:57.252787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7020f3-c8ca-43db-9189-3186eaecbd85 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:57.256689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70036939-48c9-4087-9de9-40d6d332c6b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1910803c2a18420489568bb3b2bebe64, user_id=421390792a8346a2a5ae094ee4549582, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:57.263558 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70036939-48c9-4087-9de9-40d6d332c6b0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:57.268049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db5c653-9fe0-4e08-8b0b-93a6297bd771 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:57.268684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db5c653-9fe0-4e08-8b0b-93a6297bd771 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:57.268959 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db5c653-9fe0-4e08-8b0b-93a6297bd771 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:57.269268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db5c653-9fe0-4e08-8b0b-93a6297bd771 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:57.289810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db5c653-9fe0-4e08-8b0b-93a6297bd771 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:57.299603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 881/1748] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:54:57 2022] PUT /identity/v3/projects/1910803c2a18420489568bb3b2bebe64/users/421390792a8346a2a5ae094ee4549582/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:57.301195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db5c653-9fe0-4e08-8b0b-93a6297bd771 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:57.302970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8110319-093b-4b19-8b2a-d1297cf5d72d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:57.304451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8110319-093b-4b19-8b2a-d1297cf5d72d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:57.304792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8110319-093b-4b19-8b2a-d1297cf5d72d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:57.353214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 868/1749] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:57 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:57.388776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f8110319-093b-4b19-8b2a-d1297cf5d72d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:57.397851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f8110319-093b-4b19-8b2a-d1297cf5d72d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:57.616712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 882/1750] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:57.626964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db5c653-9fe0-4e08-8b0b-93a6297bd771 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:57.784802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db5c653-9fe0-4e08-8b0b-93a6297bd771 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:57.983716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3578f550-976b-4e3b-b203-0c4909e78d0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:57.984384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578f550-976b-4e3b-b203-0c4909e78d0c service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:57.986456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578f550-976b-4e3b-b203-0c4909e78d0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:57.986818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3578f550-976b-4e3b-b203-0c4909e78d0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:58.038933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3578f550-976b-4e3b-b203-0c4909e78d0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:58.040747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3578f550-976b-4e3b-b203-0c4909e78d0c service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:58.091524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 869/1751] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:58.173996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b8a7cc-ca11-4afc-8cb6-c59ada3c6aa2 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:58.174307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b8a7cc-ca11-4afc-8cb6-c59ada3c6aa2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:58.174666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b8a7cc-ca11-4afc-8cb6-c59ada3c6aa2 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:58.176463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 883/1752] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:54:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:54:58.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f349006-8431-4a18-926f-b9bbb5a25640 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:58.290291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f349006-8431-4a18-926f-b9bbb5a25640 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:58.290515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f349006-8431-4a18-926f-b9bbb5a25640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:58.351988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-6f349006-8431-4a18-926f-b9bbb5a25640 None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:58.358150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6f349006-8431-4a18-926f-b9bbb5a25640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:58.523945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b8a7cc-ca11-4afc-8cb6-c59ada3c6aa2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:58.562784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b8a7cc-ca11-4afc-8cb6-c59ada3c6aa2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:58.579313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:58.581123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:58.581401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:58.582022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] PATH_INFO: `/v3/users/eb09b2d131cf45ef8ae8ade4b101f710` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:58.589097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb09b2d131cf45ef8ae8ade4b101f710)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:58.603536 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:58.603663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:58.674119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 870/1753] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:54:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 385 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:58.745087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb09b2d131cf45ef8ae8ade4b101f710'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:58.758505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb09b2d131cf45ef8ae8ade4b101f710'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:58.760956 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb09b2d131cf45ef8ae8ade4b101f710'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:58.762316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb09b2d131cf45ef8ae8ade4b101f710'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:58.776193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb09b2d131cf45ef8ae8ade4b101f710'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:54:58.821175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:54:58.837214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 884/1754] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:54:58 2022] DELETE /identity/v3/users/eb09b2d131cf45ef8ae8ade4b101f710 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:54:58.842626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70207d-428a-4668-ab80-479ed4d74011 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:58.843492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70207d-428a-4668-ab80-479ed4d74011 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:58.843492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70207d-428a-4668-ab80-479ed4d74011 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:58.896919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ed70207d-428a-4668-ab80-479ed4d74011 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:54:58.899737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ed70207d-428a-4668-ab80-479ed4d74011 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:59.109427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 871/1755] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:54:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:54:59.240241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:59.241713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:59.403271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1533fd66-55f4-4dbe-a0af-b42615de68b5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:59.417152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dbcc30-5f45-4e55-ab17-8d30414ed41a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:59.424959 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dbcc30-5f45-4e55-ab17-8d30414ed41a service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:59.425411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dbcc30-5f45-4e55-ab17-8d30414ed41a service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:59.425830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dbcc30-5f45-4e55-ab17-8d30414ed41a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:59.459875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1dbcc30-5f45-4e55-ab17-8d30414ed41a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:59.462458 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1dbcc30-5f45-4e55-ab17-8d30414ed41a service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:59.493233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 885/1756] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:54:59.596287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed70207d-428a-4668-ab80-479ed4d74011 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:54:59.598689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ed70207d-428a-4668-ab80-479ed4d74011 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:59.807230 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed70207d-428a-4668-ab80-479ed4d74011 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:54:59.825104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b354e224-8d53-4478-b186-78c6a42abdde service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:54:59.825896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b354e224-8d53-4478-b186-78c6a42abdde service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:54:59.828432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b354e224-8d53-4478-b186-78c6a42abdde service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:54:59.828843 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b354e224-8d53-4478-b186-78c6a42abdde service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:54:59.836963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b354e224-8d53-4478-b186-78c6a42abdde service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:54:59.889497 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b354e224-8d53-4478-b186-78c6a42abdde service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:54:59.893421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b354e224-8d53-4478-b186-78c6a42abdde service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:54:59.943099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 872/1757] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:54:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:00.337843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dbcc30-5f45-4e55-ab17-8d30414ed41a service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:00.339829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e1dbcc30-5f45-4e55-ab17-8d30414ed41a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:00.366037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dbcc30-5f45-4e55-ab17-8d30414ed41a service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:00.389849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:00.390910 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:00.394660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:00.395431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] PATH_INFO: `/v3/projects/aa0e51fa2dfe4f6f8f4e4fab437cb8b8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:00.399565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa0e51fa2dfe4f6f8f4e4fab437cb8b8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:00.402445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:00.547491 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa0e51fa2dfe4f6f8f4e4fab437cb8b8', 'request_id': 'req-361f2d11-79e0-4d93-bf4a-876752c2963a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:00.561998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa0e51fa2dfe4f6f8f4e4fab437cb8b8', 'request_id': 'req-361f2d11-79e0-4d93-bf4a-876752c2963a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:00.573751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa0e51fa2dfe4f6f8f4e4fab437cb8b8', 'request_id': 'req-361f2d11-79e0-4d93-bf4a-876752c2963a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:00.594732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] The token cache is being invalidate because project aa0e51fa2dfe4f6f8f4e4fab437cb8b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:00.595113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:00.596364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:00.598957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 886/1758] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:00 2022] DELETE /identity/v3/projects/aa0e51fa2dfe4f6f8f4e4fab437cb8b8 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:00.602785 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b354e224-8d53-4478-b186-78c6a42abdde service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:00.644384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b354e224-8d53-4478-b186-78c6a42abdde service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:00.912523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:00.914433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:00.914876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:00.915319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] PATH_INFO: `/v3/users/1748baf6a8854cbe82ba8524994b18ad` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:00.916836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1748baf6a8854cbe82ba8524994b18ad)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:00.919585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:00.978450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:01.003766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:01.005868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:01.039800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f2d11-79e0-4d93-bf4a-876752c2963a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:01.051325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1748baf6a8854cbe82ba8524994b18ad'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.052622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1748baf6a8854cbe82ba8524994b18ad'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.058448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:01.059171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:01.059448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:01.059713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] PATH_INFO: `/v3/projects/0010f98a56a04cbfbaafdfe771287dce` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:01.065502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0010f98a56a04cbfbaafdfe771287dce)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:01.075059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1748baf6a8854cbe82ba8524994b18ad'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.075794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1748baf6a8854cbe82ba8524994b18ad'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.085647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:01.086642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1748baf6a8854cbe82ba8524994b18ad'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.125479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 873/1759] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:00 2022] DELETE /identity/v3/users/1748baf6a8854cbe82ba8524994b18ad => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:01.238901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0010f98a56a04cbfbaafdfe771287dce', 'request_id': 'req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.287606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0010f98a56a04cbfbaafdfe771287dce', 'request_id': 'req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.296980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0010f98a56a04cbfbaafdfe771287dce', 'request_id': 'req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.313544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] The token cache is being invalidate because project 0010f98a56a04cbfbaafdfe771287dce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:01.314233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.315153 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.318879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 887/1760] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:01 2022] DELETE /identity/v3/projects/0010f98a56a04cbfbaafdfe771287dce => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:01.323140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:01.324600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:01.539423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb45edfd-edcd-465b-b2a1-69be92ec1979 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:01.727284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:01.732257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:01.734291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:01.735048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] PATH_INFO: `/v3/users/8b72239238f54eef8fe1759067773736` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:01.741782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b72239238f54eef8fe1759067773736)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:01.745962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:01.938352 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b72239238f54eef8fe1759067773736'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.940085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b72239238f54eef8fe1759067773736'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.943255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:01.962148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b72239238f54eef8fe1759067773736'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.962890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b72239238f54eef8fe1759067773736'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:01.984744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b72239238f54eef8fe1759067773736'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.106786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 874/1761] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:01 2022] DELETE /identity/v3/users/8b72239238f54eef8fe1759067773736 => generated 0 bytes in 784 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:02.233615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:02.235791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:02.272681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:02.282176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b90d0b5-d77c-4e05-8fe2-fbe73a955919 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:02.458626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:02.459276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:02.459555 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:02.463797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] PATH_INFO: `/v3/projects/21effdef168e4e4ab8e5fffcdb4c9c95` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:02.465294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=21effdef168e4e4ab8e5fffcdb4c9c95)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:02.473024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:02.551866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21effdef168e4e4ab8e5fffcdb4c9c95', 'request_id': 'req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.566796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21effdef168e4e4ab8e5fffcdb4c9c95', 'request_id': 'req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.591113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21effdef168e4e4ab8e5fffcdb4c9c95', 'request_id': 'req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.622368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] The token cache is being invalidate because project 21effdef168e4e4ab8e5fffcdb4c9c95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:02.622368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.625472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.628446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 888/1762] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:02 2022] DELETE /identity/v3/projects/21effdef168e4e4ab8e5fffcdb4c9c95 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:02.633154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:02.634613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:02.655412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2543460-4e1b-4518-a92c-4c878964f83c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:02.794672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:02.795331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:02.795616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:02.795885 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] PATH_INFO: `/v3/users/0b5649b0230449288817788f3d4f874f` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:02.797240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b5649b0230449288817788f3d4f874f)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:02.799291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:02.916377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b5649b0230449288817788f3d4f874f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.917468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b5649b0230449288817788f3d4f874f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.927177 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b5649b0230449288817788f3d4f874f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.928355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b5649b0230449288817788f3d4f874f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.943015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f1b41714-bf65-487e-ad12-5fc78156f144 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b5649b0230449288817788f3d4f874f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:02.972900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 875/1763] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:02 2022] DELETE /identity/v3/users/0b5649b0230449288817788f3d4f874f => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:03.881871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:03.884143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:03.892396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa367a8-29ad-413d-83c5-e6aa0f1dc4a9 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:03.892985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa367a8-29ad-413d-83c5-e6aa0f1dc4a9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:03.896805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa367a8-29ad-413d-83c5-e6aa0f1dc4a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:03.975684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbf9e2-51ae-4a62-9bba-6d8382b9e2a5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:04.034104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-5fa367a8-29ad-413d-83c5-e6aa0f1dc4a9 None None] MFA Rules not processed for user `421390792a8346a2a5ae094ee4549582`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:04.035334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5fa367a8-29ad-413d-83c5-e6aa0f1dc4a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:04.179149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:04.214697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:04.215274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:04.215551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:04.215823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] PATH_INFO: `/v3/projects/1613eb1e67d342279ad00aa71293c088` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:04.219092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] RBAC: Authorizing `identity:delete_project(project_id=1613eb1e67d342279ad00aa71293c088)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:04.221069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:04.287580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 876/1764] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:03 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 396 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:04.288462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa367a8-29ad-413d-83c5-e6aa0f1dc4a9 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:04.290674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5fa367a8-29ad-413d-83c5-e6aa0f1dc4a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:04.325937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa367a8-29ad-413d-83c5-e6aa0f1dc4a9 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:04.334790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:04.355673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1613eb1e67d342279ad00aa71293c088', 'request_id': 'req-3cb8e2de-d301-4474-86b3-ddd82a2e306b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:04.374512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1613eb1e67d342279ad00aa71293c088', 'request_id': 'req-3cb8e2de-d301-4474-86b3-ddd82a2e306b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:04.386931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1613eb1e67d342279ad00aa71293c088', 'request_id': 'req-3cb8e2de-d301-4474-86b3-ddd82a2e306b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:04.413061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] The token cache is being invalidate because project 1613eb1e67d342279ad00aa71293c088 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:04.414469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:04.415822 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3cb8e2de-d301-4474-86b3-ddd82a2e306b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:04.423104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 889/1765] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:03 2022] DELETE /identity/v3/projects/1613eb1e67d342279ad00aa71293c088 => generated 0 bytes in 538 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:04.423104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c86483-1e57-4635-b037-e8156c31a42f None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:04.423104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c86483-1e57-4635-b037-e8156c31a42f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:04.424307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c86483-1e57-4635-b037-e8156c31a42f None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:04.427792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 890/1766] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:55:04 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:55:04.430789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c86483-1e57-4635-b037-e8156c31a42f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:04.435023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-81c86483-1e57-4635-b037-e8156c31a42f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:04.470763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:04.471419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:04.471703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:04.471978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] PATH_INFO: `/v3/projects/95dd8c4315154fe4980d3f4a23477823` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:04.477091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95dd8c4315154fe4980d3f4a23477823)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:04.479132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:04.630590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c86483-1e57-4635-b037-e8156c31a42f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:04.656415 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e8cfdb-7d54-4666-ab0a-a2ed18db041a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:04.657498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8cfdb-7d54-4666-ab0a-a2ed18db041a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:04.659328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8cfdb-7d54-4666-ab0a-a2ed18db041a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:04.659847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e8cfdb-7d54-4666-ab0a-a2ed18db041a admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:04.662394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e8cfdb-7d54-4666-ab0a-a2ed18db041a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:04.666660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e8cfdb-7d54-4666-ab0a-a2ed18db041a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:04.713411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 891/1767] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:04 2022] POST /identity/v3/projects => generated 403 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:04.715425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3023eed-d5bb-4a92-830e-6ee00e66f54a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:04.715783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3023eed-d5bb-4a92-830e-6ee00e66f54a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:04.717013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3023eed-d5bb-4a92-830e-6ee00e66f54a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:04.720460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95dd8c4315154fe4980d3f4a23477823', 'request_id': 'req-581cb4f4-c27d-4eb6-a455-738698760ce4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:04.761762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95dd8c4315154fe4980d3f4a23477823', 'request_id': 'req-581cb4f4-c27d-4eb6-a455-738698760ce4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:04.761955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c3023eed-d5bb-4a92-830e-6ee00e66f54a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:04.763072 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c3023eed-d5bb-4a92-830e-6ee00e66f54a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:04.770799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95dd8c4315154fe4980d3f4a23477823', 'request_id': 'req-581cb4f4-c27d-4eb6-a455-738698760ce4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:04.786300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] The token cache is being invalidate because project 95dd8c4315154fe4980d3f4a23477823 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:04.786591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:04.787379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:04.788987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 877/1768] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:04 2022] DELETE /identity/v3/projects/95dd8c4315154fe4980d3f4a23477823 => generated 0 bytes in 501 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:04.793365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:04.794832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:04.947921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581cb4f4-c27d-4eb6-a455-738698760ce4 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:04.976714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:05.010130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 892/1769] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 291 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:05.010130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b5b7dd-b6f3-4974-9a00-d7bcd8d66f81 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.010130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b5b7dd-b6f3-4974-9a00-d7bcd8d66f81 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.010130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b5b7dd-b6f3-4974-9a00-d7bcd8d66f81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.064079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-52b5b7dd-b6f3-4974-9a00-d7bcd8d66f81 None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:05.065308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-52b5b7dd-b6f3-4974-9a00-d7bcd8d66f81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:05.074314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:05.075694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.076150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.076708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] PATH_INFO: `/v3/projects/5d513486accb4b71a08a5c2d8d79ea55` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.078513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d513486accb4b71a08a5c2d8d79ea55)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:05.081959 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:05.158576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d513486accb4b71a08a5c2d8d79ea55', 'request_id': 'req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:05.181638 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d513486accb4b71a08a5c2d8d79ea55', 'request_id': 'req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:05.193786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:05.210054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d513486accb4b71a08a5c2d8d79ea55', 'request_id': 'req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:05.235600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] The token cache is being invalidate because project 5d513486accb4b71a08a5c2d8d79ea55 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:05.236942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:05.239544 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:05.244280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 878/1770] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:04 2022] DELETE /identity/v3/projects/5d513486accb4b71a08a5c2d8d79ea55 => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:05.245795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:05.250331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:05.294925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 893/1771] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:55:05 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 290 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:05.297332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebe4e56-b0dd-43e9-a7a2-76794f0e1ea5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.298395 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebe4e56-b0dd-43e9-a7a2-76794f0e1ea5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.298902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebe4e56-b0dd-43e9-a7a2-76794f0e1ea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.361224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8ebe4e56-b0dd-43e9-a7a2-76794f0e1ea5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:05.362546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8ebe4e56-b0dd-43e9-a7a2-76794f0e1ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:05.446895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5f1bd0-0641-4851-a284-9a0aecdf0c96 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:05.452153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106ad275-b680-4073-a5ce-d63ac01169d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:05.452763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106ad275-b680-4073-a5ce-d63ac01169d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.453208 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106ad275-b680-4073-a5ce-d63ac01169d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.453422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106ad275-b680-4073-a5ce-d63ac01169d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.454678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106ad275-b680-4073-a5ce-d63ac01169d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:05.456468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106ad275-b680-4073-a5ce-d63ac01169d5 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:05.485318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 879/1772] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:05 2022] POST /identity/v3/users => generated 308 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:05.486682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c9ed36-8b56-45bd-abb5-bea0ad6f9ed6 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.486992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c9ed36-8b56-45bd-abb5-bea0ad6f9ed6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.487251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c9ed36-8b56-45bd-abb5-bea0ad6f9ed6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.502241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 894/1773] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:05.502241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09813d5-a999-4b7b-af6f-b596df1f14d0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.502241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09813d5-a999-4b7b-af6f-b596df1f14d0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.502241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09813d5-a999-4b7b-af6f-b596df1f14d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.511825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-65c9ed36-8b56-45bd-abb5-bea0ad6f9ed6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:05.513097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-65c9ed36-8b56-45bd-abb5-bea0ad6f9ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:05.522038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c09813d5-a999-4b7b-af6f-b596df1f14d0 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:05.523377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c09813d5-a999-4b7b-af6f-b596df1f14d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:05.524634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 880/1774] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:05.526932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c9ed36-8b56-45bd-abb5-bea0ad6f9ed6 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:05.528187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-65c9ed36-8b56-45bd-abb5-bea0ad6f9ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:05.534254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 895/1775] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:05.535283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09813d5-a999-4b7b-af6f-b596df1f14d0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:05.545233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09813d5-a999-4b7b-af6f-b596df1f14d0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:05.550112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a53c6e0-6beb-464b-8299-ef3d41b2a73e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:05.550708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a53c6e0-6beb-464b-8299-ef3d41b2a73e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.550982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a53c6e0-6beb-464b-8299-ef3d41b2a73e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.551249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a53c6e0-6beb-464b-8299-ef3d41b2a73e admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.552547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a53c6e0-6beb-464b-8299-ef3d41b2a73e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:05.554164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a53c6e0-6beb-464b-8299-ef3d41b2a73e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:05.563777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:05.568849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 896/1776] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:05.568849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16565de6-718b-41c3-9e32-b0478d3f1d60 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.568849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16565de6-718b-41c3-9e32-b0478d3f1d60 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.568849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16565de6-718b-41c3-9e32-b0478d3f1d60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.591414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-16565de6-718b-41c3-9e32-b0478d3f1d60 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:05.592622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-16565de6-718b-41c3-9e32-b0478d3f1d60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:05.602136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 897/1777] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:05.603008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16565de6-718b-41c3-9e32-b0478d3f1d60 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:05.615629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16565de6-718b-41c3-9e32-b0478d3f1d60 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:05.620368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c9ed36-8b56-45bd-abb5-bea0ad6f9ed6 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:05.623126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b51293-fcb2-4c4e-9409-1cb1d3bccffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:05.623740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b51293-fcb2-4c4e-9409-1cb1d3bccffb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.624014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b51293-fcb2-4c4e-9409-1cb1d3bccffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.624274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b51293-fcb2-4c4e-9409-1cb1d3bccffb admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.624912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b51293-fcb2-4c4e-9409-1cb1d3bccffb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:05.625237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b51293-fcb2-4c4e-9409-1cb1d3bccffb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:05.626449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265590fa-c80b-427a-b71f-36a5faa702fc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:05.626827 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b51293-fcb2-4c4e-9409-1cb1d3bccffb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:05.627062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265590fa-c80b-427a-b71f-36a5faa702fc service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.627342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265590fa-c80b-427a-b71f-36a5faa702fc service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.627616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265590fa-c80b-427a-b71f-36a5faa702fc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.639181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265590fa-c80b-427a-b71f-36a5faa702fc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:05.641821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265590fa-c80b-427a-b71f-36a5faa702fc service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:05.645456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 898/1778] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:05.646753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7126da-f229-4be8-a050-fcea0e23ea63 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.647034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7126da-f229-4be8-a050-fcea0e23ea63 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.647405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7126da-f229-4be8-a050-fcea0e23ea63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.655964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 881/1779] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:05.667107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265590fa-c80b-427a-b71f-36a5faa702fc service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:05.673055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-db7126da-f229-4be8-a050-fcea0e23ea63 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:05.674619 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-db7126da-f229-4be8-a050-fcea0e23ea63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:05.682549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265590fa-c80b-427a-b71f-36a5faa702fc service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:05.687222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32052c02-2122-49a6-88cf-5389478647a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:05.687875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32052c02-2122-49a6-88cf-5389478647a8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.688153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32052c02-2122-49a6-88cf-5389478647a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.688446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32052c02-2122-49a6-88cf-5389478647a8 admin admin] PATH_INFO: `/v3/projects/9925b233462a4323809ececfa452ded7/users/cb05687996e444168fe32083fc8c9a53/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.691129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 899/1780] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:05.692110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7126da-f229-4be8-a050-fcea0e23ea63 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:05.701868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7126da-f229-4be8-a050-fcea0e23ea63 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:05.706565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32052c02-2122-49a6-88cf-5389478647a8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9925b233462a4323809ececfa452ded7, user_id=cb05687996e444168fe32083fc8c9a53, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:05.706687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830ceb3c-1e50-42c1-acd8-3b8733a916db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:05.707448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830ceb3c-1e50-42c1-acd8-3b8733a916db admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.707902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830ceb3c-1e50-42c1-acd8-3b8733a916db admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.708514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830ceb3c-1e50-42c1-acd8-3b8733a916db admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.709236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830ceb3c-1e50-42c1-acd8-3b8733a916db admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:05.709354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32052c02-2122-49a6-88cf-5389478647a8 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:05.710094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830ceb3c-1e50-42c1-acd8-3b8733a916db admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:05.711496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830ceb3c-1e50-42c1-acd8-3b8733a916db admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:05.729098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 882/1781] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:05 2022] PUT /identity/v3/projects/9925b233462a4323809ececfa452ded7/users/cb05687996e444168fe32083fc8c9a53/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:05.730159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32052c02-2122-49a6-88cf-5389478647a8 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:05.731457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 900/1782] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:05.732986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830ceb3c-1e50-42c1-acd8-3b8733a916db admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:05.745471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830ceb3c-1e50-42c1-acd8-3b8733a916db admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:05.820035 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32052c02-2122-49a6-88cf-5389478647a8 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:05.833666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7693814f-4c3d-4405-9ef6-a2459b631f60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:05.834488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7693814f-4c3d-4405-9ef6-a2459b631f60 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.834780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7693814f-4c3d-4405-9ef6-a2459b631f60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.835057 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7693814f-4c3d-4405-9ef6-a2459b631f60 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.835702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7693814f-4c3d-4405-9ef6-a2459b631f60 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:05.838013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7693814f-4c3d-4405-9ef6-a2459b631f60 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:05.868687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05577e22-c542-473b-824a-cf6476b1170f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:05.869315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05577e22-c542-473b-824a-cf6476b1170f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.869591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05577e22-c542-473b-824a-cf6476b1170f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.869862 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05577e22-c542-473b-824a-cf6476b1170f admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.870540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05577e22-c542-473b-824a-cf6476b1170f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:05.870839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05577e22-c542-473b-824a-cf6476b1170f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:05.882724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05577e22-c542-473b-824a-cf6476b1170f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:05.899812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 901/1783] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:05.900686 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05577e22-c542-473b-824a-cf6476b1170f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:05.903846 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 883/1784] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:05 2022] POST /identity/v3/projects => generated 375 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:05.904718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7693814f-4c3d-4405-9ef6-a2459b631f60 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:05.919444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05577e22-c542-473b-824a-cf6476b1170f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:05.939754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:05.989441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e163ad8-5e60-475e-bffc-674545d0d5e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:05.990349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e163ad8-5e60-475e-bffc-674545d0d5e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:05.990831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e163ad8-5e60-475e-bffc-674545d0d5e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:05.991243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e163ad8-5e60-475e-bffc-674545d0d5e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:05.991614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7693814f-4c3d-4405-9ef6-a2459b631f60 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:05.992579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e163ad8-5e60-475e-bffc-674545d0d5e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:05.994361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e163ad8-5e60-475e-bffc-674545d0d5e3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:05.998747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb7c254-0bcd-447a-95b7-d13e13d3e88d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.000111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb7c254-0bcd-447a-95b7-d13e13d3e88d admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.000503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb7c254-0bcd-447a-95b7-d13e13d3e88d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.000895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb7c254-0bcd-447a-95b7-d13e13d3e88d admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.001780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb7c254-0bcd-447a-95b7-d13e13d3e88d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.004140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb7c254-0bcd-447a-95b7-d13e13d3e88d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.007464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 902/1785] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:05 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:06.008597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e163ad8-5e60-475e-bffc-674545d0d5e3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.032816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e163ad8-5e60-475e-bffc-674545d0d5e3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.041897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10d6634-2043-4f0f-b176-e8381717cca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.042919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d6634-2043-4f0f-b176-e8381717cca7 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.043384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d6634-2043-4f0f-b176-e8381717cca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.044013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10d6634-2043-4f0f-b176-e8381717cca7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.044872 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10d6634-2043-4f0f-b176-e8381717cca7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.047279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10d6634-2043-4f0f-b176-e8381717cca7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.054938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:06.084061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 884/1786] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:05 2022] POST /identity/v3/projects => generated 385 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:06.086800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb7c254-0bcd-447a-95b7-d13e13d3e88d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.091288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 903/1787] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:06 2022] POST /identity/v3/projects => generated 381 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:06.092405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10d6634-2043-4f0f-b176-e8381717cca7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.210730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb7c254-0bcd-447a-95b7-d13e13d3e88d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.212981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10d6634-2043-4f0f-b176-e8381717cca7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.223899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cf5fa4-98d7-4eb3-9c16-7d3a78d83950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.224072 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab22616-81b1-4db4-9aa0-7848e160695a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.224617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab22616-81b1-4db4-9aa0-7848e160695a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.224720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cf5fa4-98d7-4eb3-9c16-7d3a78d83950 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.225307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab22616-81b1-4db4-9aa0-7848e160695a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.225429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cf5fa4-98d7-4eb3-9c16-7d3a78d83950 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.225519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab22616-81b1-4db4-9aa0-7848e160695a admin admin] PATH_INFO: `/v3/projects/9925b233462a4323809ececfa452ded7/users/cb05687996e444168fe32083fc8c9a53/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.225619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49cf5fa4-98d7-4eb3-9c16-7d3a78d83950 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.226227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cf5fa4-98d7-4eb3-9c16-7d3a78d83950 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.227199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab22616-81b1-4db4-9aa0-7848e160695a admin admin] RBAC: Authorizing `identity:create_grant(project_id=9925b233462a4323809ececfa452ded7, user_id=cb05687996e444168fe32083fc8c9a53, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.228456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49cf5fa4-98d7-4eb3-9c16-7d3a78d83950 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.230391 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab22616-81b1-4db4-9aa0-7848e160695a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.242697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 904/1788] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:06 2022] PUT /identity/v3/projects/9925b233462a4323809ececfa452ded7/users/cb05687996e444168fe32083fc8c9a53/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:06.244047 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab22616-81b1-4db4-9aa0-7848e160695a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.258506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 885/1789] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:06 2022] POST /identity/v3/users => generated 288 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:06.259336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cf5fa4-98d7-4eb3-9c16-7d3a78d83950 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.272772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49cf5fa4-98d7-4eb3-9c16-7d3a78d83950 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.348578 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab22616-81b1-4db4-9aa0-7848e160695a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.363166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359ea6a5-db14-4db4-b229-35af369d7081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.364905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ea6a5-db14-4db4-b229-35af369d7081 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.365260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ea6a5-db14-4db4-b229-35af369d7081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.365704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ea6a5-db14-4db4-b229-35af369d7081 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.367053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359ea6a5-db14-4db4-b229-35af369d7081 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.367805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23f4868-e8e2-43f6-943f-fa4d4f8047b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.370508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359ea6a5-db14-4db4-b229-35af369d7081 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.370653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f4868-e8e2-43f6-943f-fa4d4f8047b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.371169 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f4868-e8e2-43f6-943f-fa4d4f8047b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.372560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f4868-e8e2-43f6-943f-fa4d4f8047b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.373283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23f4868-e8e2-43f6-943f-fa4d4f8047b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.378837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23f4868-e8e2-43f6-943f-fa4d4f8047b3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.413617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 886/1790] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:06 2022] POST /identity/v3/users => generated 291 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:06.413617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359ea6a5-db14-4db4-b229-35af369d7081 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.423312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359ea6a5-db14-4db4-b229-35af369d7081 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.429285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 905/1791] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:06 2022] POST /identity/v3/users => generated 293 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:06.429285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23f4868-e8e2-43f6-943f-fa4d4f8047b3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.430277 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa71f43-7c40-4841-aece-fefeac53908b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.430431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa71f43-7c40-4841-aece-fefeac53908b admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.430693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa71f43-7c40-4841-aece-fefeac53908b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.430990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa71f43-7c40-4841-aece-fefeac53908b admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.431662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa71f43-7c40-4841-aece-fefeac53908b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.433412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa71f43-7c40-4841-aece-fefeac53908b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.443698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23f4868-e8e2-43f6-943f-fa4d4f8047b3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.449834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74abb145-a408-47b3-8761-fa3f88492525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.450044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 887/1792] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:06 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:06.450747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74abb145-a408-47b3-8761-fa3f88492525 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.450862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa71f43-7c40-4841-aece-fefeac53908b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.451415 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74abb145-a408-47b3-8761-fa3f88492525 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.451690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74abb145-a408-47b3-8761-fa3f88492525 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.452455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74abb145-a408-47b3-8761-fa3f88492525 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.454404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74abb145-a408-47b3-8761-fa3f88492525 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.463081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa71f43-7c40-4841-aece-fefeac53908b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.469354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb2a6b3-ecdf-48a1-9b55-edd6a8eaa192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.470015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb2a6b3-ecdf-48a1-9b55-edd6a8eaa192 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.470370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb2a6b3-ecdf-48a1-9b55-edd6a8eaa192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.470651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb2a6b3-ecdf-48a1-9b55-edd6a8eaa192 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.471351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb2a6b3-ecdf-48a1-9b55-edd6a8eaa192 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.473050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb2a6b3-ecdf-48a1-9b55-edd6a8eaa192 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.477205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 906/1793] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:06 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:06.478383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74abb145-a408-47b3-8761-fa3f88492525 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.490984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 888/1794] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:06 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:06.490984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb2a6b3-ecdf-48a1-9b55-edd6a8eaa192 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.497011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74abb145-a408-47b3-8761-fa3f88492525 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.499499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb2a6b3-ecdf-48a1-9b55-edd6a8eaa192 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.504026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b186e78-6a65-4dbe-85f4-21978116d8f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.504933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b186e78-6a65-4dbe-85f4-21978116d8f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.505344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b186e78-6a65-4dbe-85f4-21978116d8f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.505872 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b186e78-6a65-4dbe-85f4-21978116d8f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.505990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811c8863-bdaa-4020-841d-cef4b065e545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.506645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811c8863-bdaa-4020-841d-cef4b065e545 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.506907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811c8863-bdaa-4020-841d-cef4b065e545 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.507599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b186e78-6a65-4dbe-85f4-21978116d8f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.507719 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811c8863-bdaa-4020-841d-cef4b065e545 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.508403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811c8863-bdaa-4020-841d-cef4b065e545 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.509844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b186e78-6a65-4dbe-85f4-21978116d8f4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.510340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811c8863-bdaa-4020-841d-cef4b065e545 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.511629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:06.529616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 907/1795] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:06 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:06.531348 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b186e78-6a65-4dbe-85f4-21978116d8f4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.543134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b186e78-6a65-4dbe-85f4-21978116d8f4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.548852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138f6d3c-b820-474a-b871-a73270ec3ba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.549628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f6d3c-b820-474a-b871-a73270ec3ba6 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.550547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f6d3c-b820-474a-b871-a73270ec3ba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.550547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f6d3c-b820-474a-b871-a73270ec3ba6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.550876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138f6d3c-b820-474a-b871-a73270ec3ba6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.553441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138f6d3c-b820-474a-b871-a73270ec3ba6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.561239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 889/1796] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:06 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:06.561239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811c8863-bdaa-4020-841d-cef4b065e545 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.572972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 908/1797] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:06 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:06.573908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138f6d3c-b820-474a-b871-a73270ec3ba6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.583632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811c8863-bdaa-4020-841d-cef4b065e545 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.587158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138f6d3c-b820-474a-b871-a73270ec3ba6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.592272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb82c84-a50a-4cc3-b29d-4f022e4a4fea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.593415 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb82c84-a50a-4cc3-b29d-4f022e4a4fea admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.593698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb82c84-a50a-4cc3-b29d-4f022e4a4fea admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.593972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb82c84-a50a-4cc3-b29d-4f022e4a4fea admin admin] PATH_INFO: `/v3/projects/bcaa180307564dcea9a31d51535e4dc2/users/1c9bfdce013b4808aa699705915a2c96/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.599069 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a24887-35b8-47c1-b700-a05ad0f7ba9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.599767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a24887-35b8-47c1-b700-a05ad0f7ba9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.600052 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a24887-35b8-47c1-b700-a05ad0f7ba9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.600326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a24887-35b8-47c1-b700-a05ad0f7ba9e admin admin] PATH_INFO: `/v3/projects/69f68a58495d4c99820e8f37497964b9/users/b6672b9ee1e142ee880c06bcf6fa9058/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.613218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb82c84-a50a-4cc3-b29d-4f022e4a4fea admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcaa180307564dcea9a31d51535e4dc2, user_id=1c9bfdce013b4808aa699705915a2c96, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.618901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb82c84-a50a-4cc3-b29d-4f022e4a4fea admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.630213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a24887-35b8-47c1-b700-a05ad0f7ba9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=69f68a58495d4c99820e8f37497964b9, user_id=b6672b9ee1e142ee880c06bcf6fa9058, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.634460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a24887-35b8-47c1-b700-a05ad0f7ba9e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.641493 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 909/1798] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:06 2022] PUT /identity/v3/projects/bcaa180307564dcea9a31d51535e4dc2/users/1c9bfdce013b4808aa699705915a2c96/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:06.643095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb82c84-a50a-4cc3-b29d-4f022e4a4fea admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.677432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 890/1799] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:06 2022] PUT /identity/v3/projects/69f68a58495d4c99820e8f37497964b9/users/b6672b9ee1e142ee880c06bcf6fa9058/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:06.678797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d035303-41cb-4870-afac-555fadcd6969 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.679123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d035303-41cb-4870-afac-555fadcd6969 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.679378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d035303-41cb-4870-afac-555fadcd6969 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.749358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-2d035303-41cb-4870-afac-555fadcd6969 None None] MFA Rules not processed for user `1c9bfdce013b4808aa699705915a2c96`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:06.749520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb82c84-a50a-4cc3-b29d-4f022e4a4fea admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.750715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2d035303-41cb-4870-afac-555fadcd6969 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:06.756731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:06.812394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b93bda-9e87-4388-9f09-c37fe08bd0ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.813073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b93bda-9e87-4388-9f09-c37fe08bd0ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.813506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b93bda-9e87-4388-9f09-c37fe08bd0ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.813850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b93bda-9e87-4388-9f09-c37fe08bd0ac admin admin] PATH_INFO: `/v3/projects/848c0e9f7b164a02941f8141c2f79fd9/users/8a4d787cbeb64c3186864b67f2a168c6/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.831449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b93bda-9e87-4388-9f09-c37fe08bd0ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=848c0e9f7b164a02941f8141c2f79fd9, user_id=8a4d787cbeb64c3186864b67f2a168c6, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.834982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b93bda-9e87-4388-9f09-c37fe08bd0ac admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.859316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 910/1800] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:06 2022] PUT /identity/v3/projects/848c0e9f7b164a02941f8141c2f79fd9/users/8a4d787cbeb64c3186864b67f2a168c6/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:06.860423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ee686-1884-4868-96cc-5d44315a0fdc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.860702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ee686-1884-4868-96cc-5d44315a0fdc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.860971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0ee686-1884-4868-96cc-5d44315a0fdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.861527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 891/1801] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:06 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:06.865718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d035303-41cb-4870-afac-555fadcd6969 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:06.878731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d035303-41cb-4870-afac-555fadcd6969 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:06.905154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6d0ee686-1884-4868-96cc-5d44315a0fdc None None] MFA Rules not processed for user `b6672b9ee1e142ee880c06bcf6fa9058`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:06.906573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6d0ee686-1884-4868-96cc-5d44315a0fdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:06.955907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb175458-e31e-43e4-8273-b4a50e9c74fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:06.956523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb175458-e31e-43e4-8273-b4a50e9c74fb admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.956806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb175458-e31e-43e4-8273-b4a50e9c74fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.957071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb175458-e31e-43e4-8273-b4a50e9c74fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.958748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb175458-e31e-43e4-8273-b4a50e9c74fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:06.960362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:06.961290 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb175458-e31e-43e4-8273-b4a50e9c74fb admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:06.992456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 911/1802] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:06.993784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bd7fb8-9510-48f0-ad29-1bc0600bb229 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:06.994105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bd7fb8-9510-48f0-ad29-1bc0600bb229 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:06.994387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bd7fb8-9510-48f0-ad29-1bc0600bb229 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:06.995991 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 892/1803] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:06 2022] POST /identity/v3/projects => generated 379 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:06.997469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb175458-e31e-43e4-8273-b4a50e9c74fb admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.036680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-04bd7fb8-9510-48f0-ad29-1bc0600bb229 None None] MFA Rules not processed for user `8a4d787cbeb64c3186864b67f2a168c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:07.038565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-04bd7fb8-9510-48f0-ad29-1bc0600bb229 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:07.077275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb175458-e31e-43e4-8273-b4a50e9c74fb admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:07.083511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3448ac33-4b1e-4af4-b97f-244f9f0a81e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:07.084003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3448ac33-4b1e-4af4-b97f-244f9f0a81e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:07.084323 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3448ac33-4b1e-4af4-b97f-244f9f0a81e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:07.084850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3448ac33-4b1e-4af4-b97f-244f9f0a81e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:07.085310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3448ac33-4b1e-4af4-b97f-244f9f0a81e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:07.087254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3448ac33-4b1e-4af4-b97f-244f9f0a81e8 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:07.119357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 912/1804] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:07.120534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bd7fb8-9510-48f0-ad29-1bc0600bb229 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.122229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 893/1805] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:06 2022] POST /identity/v3/projects => generated 375 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:07.124171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3448ac33-4b1e-4af4-b97f-244f9f0a81e8 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.139200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bd7fb8-9510-48f0-ad29-1bc0600bb229 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:07.155968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:07.209443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3448ac33-4b1e-4af4-b97f-244f9f0a81e8 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:07.215639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed80a42-3578-4962-98ff-989014dd8f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:07.216870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed80a42-3578-4962-98ff-989014dd8f16 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:07.217396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed80a42-3578-4962-98ff-989014dd8f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:07.217690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed80a42-3578-4962-98ff-989014dd8f16 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:07.218387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed80a42-3578-4962-98ff-989014dd8f16 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:07.220370 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c7e7d9-d5a8-4967-8ffa-404acf222358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:07.221150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c7e7d9-d5a8-4967-8ffa-404acf222358 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:07.221770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c7e7d9-d5a8-4967-8ffa-404acf222358 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:07.222131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c7e7d9-d5a8-4967-8ffa-404acf222358 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:07.222854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c7e7d9-d5a8-4967-8ffa-404acf222358 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:07.226326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c7e7d9-d5a8-4967-8ffa-404acf222358 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:07.231109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed80a42-3578-4962-98ff-989014dd8f16 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:07.271497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 913/1806] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:07 2022] POST /identity/v3/users => generated 296 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:07.271497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c7e7d9-d5a8-4967-8ffa-404acf222358 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.291528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c7e7d9-d5a8-4967-8ffa-404acf222358 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:07.297208 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 894/1807] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:07 2022] POST /identity/v3/users => generated 294 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:07.297208 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed80a42-3578-4962-98ff-989014dd8f16 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.303607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd23eee-985f-4ce9-b7a1-637b4e46ff4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:07.307437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd23eee-985f-4ce9-b7a1-637b4e46ff4a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:07.307965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd23eee-985f-4ce9-b7a1-637b4e46ff4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:07.308397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd23eee-985f-4ce9-b7a1-637b4e46ff4a admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:07.310477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd23eee-985f-4ce9-b7a1-637b4e46ff4a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:07.318031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd23eee-985f-4ce9-b7a1-637b4e46ff4a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:07.322006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed80a42-3578-4962-98ff-989014dd8f16 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:07.335659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fa978d-3fcd-4b06-98e3-a798d56c76bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:07.336716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa978d-3fcd-4b06-98e3-a798d56c76bf admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:07.336999 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa978d-3fcd-4b06-98e3-a798d56c76bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:07.337314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fa978d-3fcd-4b06-98e3-a798d56c76bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:07.338003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fa978d-3fcd-4b06-98e3-a798d56c76bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:07.341196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fa978d-3fcd-4b06-98e3-a798d56c76bf admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:07.358094 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 895/1808] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:07 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:07.358094 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fa978d-3fcd-4b06-98e3-a798d56c76bf admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.369048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fa978d-3fcd-4b06-98e3-a798d56c76bf admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:07.374985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86576275-3232-430e-96a5-cee31961cbf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:07.375624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86576275-3232-430e-96a5-cee31961cbf5 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:07.375909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86576275-3232-430e-96a5-cee31961cbf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:07.376146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86576275-3232-430e-96a5-cee31961cbf5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:07.376885 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86576275-3232-430e-96a5-cee31961cbf5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:07.378541 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86576275-3232-430e-96a5-cee31961cbf5 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:07.405615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 914/1809] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:07 2022] POST /identity/v3/projects => generated 383 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:07.405615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd23eee-985f-4ce9-b7a1-637b4e46ff4a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.407416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 896/1810] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:07 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:07.412757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86576275-3232-430e-96a5-cee31961cbf5 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.433215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86576275-3232-430e-96a5-cee31961cbf5 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:07.505970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:07.541880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd23eee-985f-4ce9-b7a1-637b4e46ff4a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:07.548846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08fe1c3-bde9-4607-b111-32cac7dfcf09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:07.550916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08fe1c3-bde9-4607-b111-32cac7dfcf09 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:07.551201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08fe1c3-bde9-4607-b111-32cac7dfcf09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:07.552023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08fe1c3-bde9-4607-b111-32cac7dfcf09 admin admin] PATH_INFO: `/v3/projects/4d38869141a3496c8882848ecafbe34d/users/6c61d39b7cc540efb44f04ecbd60c42f/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:07.561262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd010110-15b8-491e-959e-894cfa189b59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:07.561883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd010110-15b8-491e-959e-894cfa189b59 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:07.562213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd010110-15b8-491e-959e-894cfa189b59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:07.562553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd010110-15b8-491e-959e-894cfa189b59 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:07.563665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd010110-15b8-491e-959e-894cfa189b59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:07.567716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd010110-15b8-491e-959e-894cfa189b59 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:07.576680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08fe1c3-bde9-4607-b111-32cac7dfcf09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d38869141a3496c8882848ecafbe34d, user_id=6c61d39b7cc540efb44f04ecbd60c42f, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:07.579864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08fe1c3-bde9-4607-b111-32cac7dfcf09 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:07.604951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 915/1811] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:07 2022] PUT /identity/v3/projects/4d38869141a3496c8882848ecafbe34d/users/6c61d39b7cc540efb44f04ecbd60c42f/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:07.604951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08fe1c3-bde9-4607-b111-32cac7dfcf09 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.614614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 897/1812] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:07 2022] POST /identity/v3/users => generated 298 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:07.616277 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de78106-3a32-4e3c-af88-c874ffd7f562 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:07.616750 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de78106-3a32-4e3c-af88-c874ffd7f562 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:07.617086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de78106-3a32-4e3c-af88-c874ffd7f562 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:07.700514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4de78106-3a32-4e3c-af88-c874ffd7f562 None None] MFA Rules not processed for user `cb05687996e444168fe32083fc8c9a53`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:07.707660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4de78106-3a32-4e3c-af88-c874ffd7f562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:07.747412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08fe1c3-bde9-4607-b111-32cac7dfcf09 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:07.759898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4425e4b3-b6bf-4f53-9d40-3194877b85b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:07.762111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4425e4b3-b6bf-4f53-9d40-3194877b85b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:07.763783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4425e4b3-b6bf-4f53-9d40-3194877b85b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:07.764907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4425e4b3-b6bf-4f53-9d40-3194877b85b5 admin admin] PATH_INFO: `/v3/projects/5f9065bedcfd4e749d70335a50a0a5f5/users/be87a99b1ede422aa402d373cb289702/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:07.834506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4425e4b3-b6bf-4f53-9d40-3194877b85b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f9065bedcfd4e749d70335a50a0a5f5, user_id=be87a99b1ede422aa402d373cb289702, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:07.847351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4425e4b3-b6bf-4f53-9d40-3194877b85b5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:07.873279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:07.890314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 898/1814] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:07 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:07.890532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 916/1814] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:07 2022] PUT /identity/v3/projects/5f9065bedcfd4e749d70335a50a0a5f5/users/be87a99b1ede422aa402d373cb289702/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:07.890532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4425e4b3-b6bf-4f53-9d40-3194877b85b5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.890692 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de78106-3a32-4e3c-af88-c874ffd7f562 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:07.916265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de78106-3a32-4e3c-af88-c874ffd7f562 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:07.990626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4425e4b3-b6bf-4f53-9d40-3194877b85b5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:08.004008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d51910-36f8-4d0e-bca0-5af8fbd9f941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.005292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d51910-36f8-4d0e-bca0-5af8fbd9f941 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.005965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d51910-36f8-4d0e-bca0-5af8fbd9f941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.006482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d51910-36f8-4d0e-bca0-5af8fbd9f941 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.008145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d51910-36f8-4d0e-bca0-5af8fbd9f941 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.014764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d51910-36f8-4d0e-bca0-5af8fbd9f941 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.037714 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 917/1815] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:07 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:08.039447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d51910-36f8-4d0e-bca0-5af8fbd9f941 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:08.058265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d51910-36f8-4d0e-bca0-5af8fbd9f941 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:08.065100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c94a90-e7d4-4300-a4ff-8042deb2e414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.066706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c94a90-e7d4-4300-a4ff-8042deb2e414 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.067333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c94a90-e7d4-4300-a4ff-8042deb2e414 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.067887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c94a90-e7d4-4300-a4ff-8042deb2e414 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.068884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c94a90-e7d4-4300-a4ff-8042deb2e414 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.069915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b8368c-6e33-4525-804d-afc59446f479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.070836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b8368c-6e33-4525-804d-afc59446f479 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.070999 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b8368c-6e33-4525-804d-afc59446f479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.071286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b8368c-6e33-4525-804d-afc59446f479 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.071911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b8368c-6e33-4525-804d-afc59446f479 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.072204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c94a90-e7d4-4300-a4ff-8042deb2e414 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.078373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b8368c-6e33-4525-804d-afc59446f479 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.089246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 918/1816] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:08 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:08.090340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c94a90-e7d4-4300-a4ff-8042deb2e414 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:08.092592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 899/1817] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:07 2022] GET /identity/v3/roles => generated 1420 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:08.093647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b8368c-6e33-4525-804d-afc59446f479 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:08.105860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b8368c-6e33-4525-804d-afc59446f479 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:08.113216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c94a90-e7d4-4300-a4ff-8042deb2e414 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:08.119327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa911f6-bb18-4938-b6f5-c493ef8f6fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.119527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b84e89-a6f8-432e-9b05-3e646697d1f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.120584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa911f6-bb18-4938-b6f5-c493ef8f6fbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.120584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa911f6-bb18-4938-b6f5-c493ef8f6fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.120762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b84e89-a6f8-432e-9b05-3e646697d1f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.120961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b84e89-a6f8-432e-9b05-3e646697d1f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.121098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa911f6-bb18-4938-b6f5-c493ef8f6fbe admin admin] PATH_INFO: `/v3/projects/5f9065bedcfd4e749d70335a50a0a5f5/users/be87a99b1ede422aa402d373cb289702/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.121207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b84e89-a6f8-432e-9b05-3e646697d1f2 admin admin] PATH_INFO: `/v3/projects/5888d84e165b4117bb88e799ce5a42f8/users/a8e0a27de1834c40abdd0619c82716f6/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.123543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa911f6-bb18-4938-b6f5-c493ef8f6fbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f9065bedcfd4e749d70335a50a0a5f5, user_id=be87a99b1ede422aa402d373cb289702, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.127383 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa911f6-bb18-4938-b6f5-c493ef8f6fbe admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.139363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b84e89-a6f8-432e-9b05-3e646697d1f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5888d84e165b4117bb88e799ce5a42f8, user_id=a8e0a27de1834c40abdd0619c82716f6, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.142505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b84e89-a6f8-432e-9b05-3e646697d1f2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.146776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 900/1818] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:08 2022] PUT /identity/v3/projects/5f9065bedcfd4e749d70335a50a0a5f5/users/be87a99b1ede422aa402d373cb289702/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:08.148678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa911f6-bb18-4938-b6f5-c493ef8f6fbe admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:08.169584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 919/1819] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:08 2022] PUT /identity/v3/projects/5888d84e165b4117bb88e799ce5a42f8/users/a8e0a27de1834c40abdd0619c82716f6/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:08.171468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f86ed0-f3d2-4456-a556-7dd77f33a422 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.172323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f86ed0-f3d2-4456-a556-7dd77f33a422 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.172923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f86ed0-f3d2-4456-a556-7dd77f33a422 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.183272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:08.208772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f9f86ed0-f3d2-4456-a556-7dd77f33a422 None None] MFA Rules not processed for user `be87a99b1ede422aa402d373cb289702`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:08.210296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f9f86ed0-f3d2-4456-a556-7dd77f33a422 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:08.244564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa911f6-bb18-4938-b6f5-c493ef8f6fbe admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:08.316595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff83d4c3-54c3-454e-8418-c78f54ec90c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.317345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83d4c3-54c3-454e-8418-c78f54ec90c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.317667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83d4c3-54c3-454e-8418-c78f54ec90c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.318014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83d4c3-54c3-454e-8418-c78f54ec90c6 admin admin] PATH_INFO: `/v3/projects/4d38869141a3496c8882848ecafbe34d/users/6c61d39b7cc540efb44f04ecbd60c42f/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.320648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff83d4c3-54c3-454e-8418-c78f54ec90c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d38869141a3496c8882848ecafbe34d, user_id=6c61d39b7cc540efb44f04ecbd60c42f, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.323459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff83d4c3-54c3-454e-8418-c78f54ec90c6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.329941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 920/1820] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:08 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:08.330707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f86ed0-f3d2-4456-a556-7dd77f33a422 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:08.343064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f86ed0-f3d2-4456-a556-7dd77f33a422 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:08.346371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 901/1821] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:08 2022] PUT /identity/v3/projects/4d38869141a3496c8882848ecafbe34d/users/6c61d39b7cc540efb44f04ecbd60c42f/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:08.351783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff83d4c3-54c3-454e-8418-c78f54ec90c6 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:08.354626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ff83d4c3-54c3-454e-8418-c78f54ec90c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:08.388097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:08.413970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7a8288-28bd-40a8-be18-f41976728cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.414947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7a8288-28bd-40a8-be18-f41976728cfa admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.414947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7a8288-28bd-40a8-be18-f41976728cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.415133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7a8288-28bd-40a8-be18-f41976728cfa admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.415888 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7a8288-28bd-40a8-be18-f41976728cfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.417918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7a8288-28bd-40a8-be18-f41976728cfa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.423544 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff83d4c3-54c3-454e-8418-c78f54ec90c6 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:08.427655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064ff226-35e7-4576-864f-2d39b4cf027a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.428679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064ff226-35e7-4576-864f-2d39b4cf027a service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.428679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064ff226-35e7-4576-864f-2d39b4cf027a service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.428857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064ff226-35e7-4576-864f-2d39b4cf027a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.432572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 921/1822] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:08 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:08.432572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73019db-ee87-4cc4-861d-c065a7c90e23 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.432572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73019db-ee87-4cc4-861d-c065a7c90e23 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.432572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73019db-ee87-4cc4-861d-c065a7c90e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.464945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b73019db-ee87-4cc4-861d-c065a7c90e23 None None] MFA Rules not processed for user `6c61d39b7cc540efb44f04ecbd60c42f`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:08.464945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b73019db-ee87-4cc4-861d-c065a7c90e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:08.498906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064ff226-35e7-4576-864f-2d39b4cf027a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.500660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064ff226-35e7-4576-864f-2d39b4cf027a service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.510256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 902/1823] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:08 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:08.511318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064ff226-35e7-4576-864f-2d39b4cf027a service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:08.527623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064ff226-35e7-4576-864f-2d39b4cf027a service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:08.534358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd4531d-d90c-4abd-a81f-34552f07a5ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.535091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd4531d-d90c-4abd-a81f-34552f07a5ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.535658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd4531d-d90c-4abd-a81f-34552f07a5ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.536398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd4531d-d90c-4abd-a81f-34552f07a5ae admin admin] PATH_INFO: `/v3/projects/5888d84e165b4117bb88e799ce5a42f8/users/a8e0a27de1834c40abdd0619c82716f6/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.538441 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd4531d-d90c-4abd-a81f-34552f07a5ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=5888d84e165b4117bb88e799ce5a42f8, user_id=a8e0a27de1834c40abdd0619c82716f6, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.541388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd4531d-d90c-4abd-a81f-34552f07a5ae admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.563149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 903/1824] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:08 2022] PUT /identity/v3/projects/5888d84e165b4117bb88e799ce5a42f8/users/a8e0a27de1834c40abdd0619c82716f6/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:08.564727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bda9a4-408c-40bc-a09e-80c9de5f3b12 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.565023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bda9a4-408c-40bc-a09e-80c9de5f3b12 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.565301 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bda9a4-408c-40bc-a09e-80c9de5f3b12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.566306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 922/1825] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:08 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:08.578297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73019db-ee87-4cc4-861d-c065a7c90e23 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:08.580053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b73019db-ee87-4cc4-861d-c065a7c90e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:08.602738 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73019db-ee87-4cc4-861d-c065a7c90e23 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:08.616831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e4bda9a4-408c-40bc-a09e-80c9de5f3b12 None None] MFA Rules not processed for user `a8e0a27de1834c40abdd0619c82716f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:08.618096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e4bda9a4-408c-40bc-a09e-80c9de5f3b12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:08.623834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:08.702149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea46c8b8-c2e7-4a54-b036-ae405a2a63eb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.703731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea46c8b8-c2e7-4a54-b036-ae405a2a63eb service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.704544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea46c8b8-c2e7-4a54-b036-ae405a2a63eb service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.705700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea46c8b8-c2e7-4a54-b036-ae405a2a63eb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.751627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 904/1826] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:08 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:08.847758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea46c8b8-c2e7-4a54-b036-ae405a2a63eb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.849791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea46c8b8-c2e7-4a54-b036-ae405a2a63eb service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.882672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 923/1827] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:08 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:08.921782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bda9a4-408c-40bc-a09e-80c9de5f3b12 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:08.933105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bda9a4-408c-40bc-a09e-80c9de5f3b12 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:08.937358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e0320f-27d3-4cab-ac61-f10c5d9aed22 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:08.938085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e0320f-27d3-4cab-ac61-f10c5d9aed22 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:08.938361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e0320f-27d3-4cab-ac61-f10c5d9aed22 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:08.938631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e0320f-27d3-4cab-ac61-f10c5d9aed22 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:08.950852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e0320f-27d3-4cab-ac61-f10c5d9aed22 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:08.952503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e0320f-27d3-4cab-ac61-f10c5d9aed22 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:08.969390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 905/1828] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:08 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:09.009125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea46c8b8-c2e7-4a54-b036-ae405a2a63eb service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:09.158935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea46c8b8-c2e7-4a54-b036-ae405a2a63eb service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:09.184787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:09.185442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.185757 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.186035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] PATH_INFO: `/v3/users/8a4d787cbeb64c3186864b67f2a168c6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.187597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a4d787cbeb64c3186864b67f2a168c6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:09.195030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:09.202736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb082bda-4f35-40da-99e2-b06d2de1771b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.203206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb082bda-4f35-40da-99e2-b06d2de1771b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.203379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb082bda-4f35-40da-99e2-b06d2de1771b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.205591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 906/1829] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:55:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:55:09.216554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad1bc00-5ba8-4387-9ac0-e63a9ec5d9cf None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.216844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad1bc00-5ba8-4387-9ac0-e63a9ec5d9cf None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.217128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad1bc00-5ba8-4387-9ac0-e63a9ec5d9cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.245326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-1ad1bc00-5ba8-4387-9ac0-e63a9ec5d9cf None None] MFA Rules not processed for user `5eeff4f989ae4338a182bfdadd3a8b63`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:09.247105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1ad1bc00-5ba8-4387-9ac0-e63a9ec5d9cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:09.275558 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a4d787cbeb64c3186864b67f2a168c6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:09.286091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a4d787cbeb64c3186864b67f2a168c6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:09.286881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a4d787cbeb64c3186864b67f2a168c6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:09.287600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a4d787cbeb64c3186864b67f2a168c6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:09.299128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:09.305059 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-68a7f5be-66c8-4800-9fa8-e6d793796f3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a4d787cbeb64c3186864b67f2a168c6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:09.307088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:09.373398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 924/1830] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:09 2022] DELETE /identity/v3/users/8a4d787cbeb64c3186864b67f2a168c6 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:09.378868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9745b06-0f14-4f8b-9d62-82a60efe2f6c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.379096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9745b06-0f14-4f8b-9d62-82a60efe2f6c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.379999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9745b06-0f14-4f8b-9d62-82a60efe2f6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.381752 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 907/1831] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:55:09 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:09.397624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7abd3fc-50ef-44bb-93be-ce073b6dd68b None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.397956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7abd3fc-50ef-44bb-93be-ce073b6dd68b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.398205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7abd3fc-50ef-44bb-93be-ce073b6dd68b None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.400019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 908/1832] 10.222.0.25 () {54 vars in 961 bytes} [Sun Oct 2 00:55:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 02 00:55:09.404651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f293f926-3358-4fd7-b4e7-39e482ad8d73 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.405001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f293f926-3358-4fd7-b4e7-39e482ad8d73 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.405196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f293f926-3358-4fd7-b4e7-39e482ad8d73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.406562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f9745b06-0f14-4f8b-9d62-82a60efe2f6c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:09.408321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f9745b06-0f14-4f8b-9d62-82a60efe2f6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:09.436315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f293f926-3358-4fd7-b4e7-39e482ad8d73 None None] MFA Rules not processed for user `5eeff4f989ae4338a182bfdadd3a8b63`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:09.437639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f293f926-3358-4fd7-b4e7-39e482ad8d73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:09.504168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 925/1833] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:09.507079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9745b06-0f14-4f8b-9d62-82a60efe2f6c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:09.508632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f9745b06-0f14-4f8b-9d62-82a60efe2f6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:09.521935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9745b06-0f14-4f8b-9d62-82a60efe2f6c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:09.523327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 909/1834] 10.222.0.25 () {58 vars in 1058 bytes} [Sun Oct 2 00:55:09 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:09.524386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f293f926-3358-4fd7-b4e7-39e482ad8d73 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:09.526377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f293f926-3358-4fd7-b4e7-39e482ad8d73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:09.527780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:09.528581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.529288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.529696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] PATH_INFO: `/v3/users/1c9bfdce013b4808aa699705915a2c96` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.531069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c9bfdce013b4808aa699705915a2c96)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:09.533245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:09.613044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c9bfdce013b4808aa699705915a2c96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:09.622537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f293f926-3358-4fd7-b4e7-39e482ad8d73 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:09.623267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c9bfdce013b4808aa699705915a2c96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:09.624178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c9bfdce013b4808aa699705915a2c96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:09.625668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c9bfdce013b4808aa699705915a2c96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:09.627945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5015e62-49df-40de-ae19-72809c280932 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:09.628744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5015e62-49df-40de-ae19-72809c280932 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.629120 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5015e62-49df-40de-ae19-72809c280932 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.629972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5015e62-49df-40de-ae19-72809c280932 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.633927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f5015e62-49df-40de-ae19-72809c280932 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:09.643208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c9bfdce013b4808aa699705915a2c96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:09.664435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5015e62-49df-40de-ae19-72809c280932 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:09.666660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5015e62-49df-40de-ae19-72809c280932 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:09.670483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 926/1835] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:09 2022] DELETE /identity/v3/users/1c9bfdce013b4808aa699705915a2c96 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:09.671409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:09.673755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:09.680842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 910/1836] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:09.682371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0c524d-479f-4e9b-b517-a71eb2b36fef None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.683019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0c524d-479f-4e9b-b517-a71eb2b36fef None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.683434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0c524d-479f-4e9b-b517-a71eb2b36fef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.707045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-cf0c524d-479f-4e9b-b517-a71eb2b36fef None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:09.712114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cf0c524d-479f-4e9b-b517-a71eb2b36fef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:09.752551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:09.778644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a852782-0ec0-4c99-8007-c69e06443467 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:09.787015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2e465a-991e-4634-8872-49f50022827c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:09.787894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2e465a-991e-4634-8872-49f50022827c service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.788442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2e465a-991e-4634-8872-49f50022827c service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.788989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2e465a-991e-4634-8872-49f50022827c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.791159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-be2e465a-991e-4634-8872-49f50022827c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:09.798385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 911/1837] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:09.805176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0c524d-479f-4e9b-b517-a71eb2b36fef None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:09.807061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cf0c524d-479f-4e9b-b517-a71eb2b36fef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:09.812470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2e465a-991e-4634-8872-49f50022827c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:09.814810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2e465a-991e-4634-8872-49f50022827c service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:09.827765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 927/1838] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:09.831227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:09.880274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0c524d-479f-4e9b-b517-a71eb2b36fef None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:09.884732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b10dc-78c3-4400-8087-391935705f9e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:09.885351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b10dc-78c3-4400-8087-391935705f9e service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:09.885629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b10dc-78c3-4400-8087-391935705f9e service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:09.885998 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b10dc-78c3-4400-8087-391935705f9e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:09.896914 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84b10dc-78c3-4400-8087-391935705f9e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:09.898090 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84b10dc-78c3-4400-8087-391935705f9e service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:09.907801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 912/1839] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:10.149567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2e465a-991e-4634-8872-49f50022827c service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:10.151293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-be2e465a-991e-4634-8872-49f50022827c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:10.171805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2e465a-991e-4634-8872-49f50022827c service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:10.181598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:10.182426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:10.186082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:10.186771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] PATH_INFO: `/v3/users/421390792a8346a2a5ae094ee4549582` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:10.190204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] RBAC: Authorizing `identity:delete_user(user_id=421390792a8346a2a5ae094ee4549582)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:10.193656 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:10.258688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '421390792a8346a2a5ae094ee4549582'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.272105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '421390792a8346a2a5ae094ee4549582'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.273111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b10dc-78c3-4400-8087-391935705f9e service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:10.274004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '421390792a8346a2a5ae094ee4549582'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.274576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c84b10dc-78c3-4400-8087-391935705f9e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:10.275735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '421390792a8346a2a5ae094ee4549582'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.285853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '421390792a8346a2a5ae094ee4549582'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.291488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b10dc-78c3-4400-8087-391935705f9e service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:10.297100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:10.297901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:10.298224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:10.298621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] PATH_INFO: `/v3/projects/848c0e9f7b164a02941f8141c2f79fd9` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:10.300807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=848c0e9f7b164a02941f8141c2f79fd9)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:10.303452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:10.323206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 928/1840] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:10 2022] DELETE /identity/v3/users/421390792a8346a2a5ae094ee4549582 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:10.324191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:10.330200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:10.377597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '848c0e9f7b164a02941f8141c2f79fd9', 'request_id': 'req-ee90e863-c3c6-4869-ac62-ffebd53043ab'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.389423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '848c0e9f7b164a02941f8141c2f79fd9', 'request_id': 'req-ee90e863-c3c6-4869-ac62-ffebd53043ab'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.397718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '848c0e9f7b164a02941f8141c2f79fd9', 'request_id': 'req-ee90e863-c3c6-4869-ac62-ffebd53043ab'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.408674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] The token cache is being invalidate because project 848c0e9f7b164a02941f8141c2f79fd9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:10.409092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.409857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.411783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 913/1841] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:10 2022] DELETE /identity/v3/projects/848c0e9f7b164a02941f8141c2f79fd9 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:10.416542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0337ac-533f-4b5b-a614-cb9948e57474 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:10.417693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:10.420817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:10.488633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:10.489708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:10.489935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee90e863-c3c6-4869-ac62-ffebd53043ab admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:10.490578 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:10.491164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] PATH_INFO: `/v3/projects/bcaa180307564dcea9a31d51535e4dc2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:10.492816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bcaa180307564dcea9a31d51535e4dc2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:10.494675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:10.495317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:10.495591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:10.495861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] PATH_INFO: `/v3/users/a8e0a27de1834c40abdd0619c82716f6` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:10.496801 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:10.497301 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8e0a27de1834c40abdd0619c82716f6)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:10.499471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:10.499921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:10.551925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:10.559265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8e0a27de1834c40abdd0619c82716f6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.559930 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8e0a27de1834c40abdd0619c82716f6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.561224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bcaa180307564dcea9a31d51535e4dc2', 'request_id': 'req-ae1e585e-d19a-4fb8-a548-430dfd3a0188'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.568403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8e0a27de1834c40abdd0619c82716f6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.569220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8e0a27de1834c40abdd0619c82716f6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.571711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bcaa180307564dcea9a31d51535e4dc2', 'request_id': 'req-ae1e585e-d19a-4fb8-a548-430dfd3a0188'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.577546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8e0a27de1834c40abdd0619c82716f6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.580462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bcaa180307564dcea9a31d51535e4dc2', 'request_id': 'req-ae1e585e-d19a-4fb8-a548-430dfd3a0188'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.593914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] The token cache is being invalidate because project bcaa180307564dcea9a31d51535e4dc2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:10.594402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.595099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.599595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 914/1842] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:10 2022] DELETE /identity/v3/users/a8e0a27de1834c40abdd0619c82716f6 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:10.602699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 929/1843] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:10 2022] DELETE /identity/v3/projects/bcaa180307564dcea9a31d51535e4dc2 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:10.606694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:10.608452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:10.700766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:10.703393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:10.736495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057babbe-eeae-4183-9a12-c87b80b02255 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:10.744578 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:10.746654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:10.747155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:10.748197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] PATH_INFO: `/v3/users/6c61d39b7cc540efb44f04ecbd60c42f` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:10.750298 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c61d39b7cc540efb44f04ecbd60c42f)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:10.754560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:10.864578 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c61d39b7cc540efb44f04ecbd60c42f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.867235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c61d39b7cc540efb44f04ecbd60c42f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.875795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1e585e-d19a-4fb8-a548-430dfd3a0188 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:10.886845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c61d39b7cc540efb44f04ecbd60c42f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.888874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c61d39b7cc540efb44f04ecbd60c42f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.889078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:10.890096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:10.890613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:10.891103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] PATH_INFO: `/v3/projects/1910803c2a18420489568bb3b2bebe64` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:10.893085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1910803c2a18420489568bb3b2bebe64)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:10.897829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:10.938407 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c61d39b7cc540efb44f04ecbd60c42f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:10.966939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 915/1844] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:10 2022] DELETE /identity/v3/users/6c61d39b7cc540efb44f04ecbd60c42f => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:11.053236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1910803c2a18420489568bb3b2bebe64', 'request_id': 'req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.082275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1910803c2a18420489568bb3b2bebe64', 'request_id': 'req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.097338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1910803c2a18420489568bb3b2bebe64', 'request_id': 'req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.115487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] The token cache is being invalidate because project 1910803c2a18420489568bb3b2bebe64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:11.116138 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.117034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.120236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 930/1845] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:10 2022] DELETE /identity/v3/projects/1910803c2a18420489568bb3b2bebe64 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:11.124360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:11.125922 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:11.158333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:11.159759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:11.206178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:11.212802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:11.258289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5aec61a-92cd-4c32-988f-c8f8329a3ee2 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:11.264044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:11.266312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:11.266929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:11.267460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] PATH_INFO: `/v3/users/cb05687996e444168fe32083fc8c9a53` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:11.268817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb05687996e444168fe32083fc8c9a53)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:11.270562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c8fb98-479b-494b-bf80-c6ca4c3b70d0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:11.272353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:11.276284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:11.279075 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:11.279470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:11.279865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] PATH_INFO: `/v3/projects/5888d84e165b4117bb88e799ce5a42f8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:11.281216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=5888d84e165b4117bb88e799ce5a42f8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:11.283368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:11.356850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5888d84e165b4117bb88e799ce5a42f8', 'request_id': 'req-570d5076-7026-402b-a766-e5da153036fd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.366162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5888d84e165b4117bb88e799ce5a42f8', 'request_id': 'req-570d5076-7026-402b-a766-e5da153036fd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.392427 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5888d84e165b4117bb88e799ce5a42f8', 'request_id': 'req-570d5076-7026-402b-a766-e5da153036fd'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.418473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] The token cache is being invalidate because project 5888d84e165b4117bb88e799ce5a42f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:11.422974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.424562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-570d5076-7026-402b-a766-e5da153036fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.425637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb05687996e444168fe32083fc8c9a53'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.427738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb05687996e444168fe32083fc8c9a53'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.433896 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 931/1846] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:11 2022] DELETE /identity/v3/projects/5888d84e165b4117bb88e799ce5a42f8 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:11.436989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960bf5d5-b71c-479e-b4b8-fbdf8c9cddaa None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:11.437377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960bf5d5-b71c-479e-b4b8-fbdf8c9cddaa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:11.437800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960bf5d5-b71c-479e-b4b8-fbdf8c9cddaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:11.446681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb05687996e444168fe32083fc8c9a53'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.447932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb05687996e444168fe32083fc8c9a53'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.480812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-960bf5d5-b71c-479e-b4b8-fbdf8c9cddaa None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:11.482788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-43a27373-7176-473d-a887-eebb98a17aa1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb05687996e444168fe32083fc8c9a53'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.483148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-960bf5d5-b71c-479e-b4b8-fbdf8c9cddaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:11.560483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 916/1847] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:11 2022] DELETE /identity/v3/users/cb05687996e444168fe32083fc8c9a53 => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:11.605344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 932/1848] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:11.607839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d33f9-a9cb-44a9-b983-a2ca792ab04b None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:11.608354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d33f9-a9cb-44a9-b983-a2ca792ab04b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:11.608856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9d33f9-a9cb-44a9-b983-a2ca792ab04b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:11.645809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-2d9d33f9-a9cb-44a9-b983-a2ca792ab04b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:11.648095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2d9d33f9-a9cb-44a9-b983-a2ca792ab04b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:11.661604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960bf5d5-b71c-479e-b4b8-fbdf8c9cddaa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:11.664256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-960bf5d5-b71c-479e-b4b8-fbdf8c9cddaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:11.692286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960bf5d5-b71c-479e-b4b8-fbdf8c9cddaa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:11.806374 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:11.819786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 917/1849] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:11.823128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:11.823674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9d33f9-a9cb-44a9-b983-a2ca792ab04b None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:11.823789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:11.824063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:11.824753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] PATH_INFO: `/v3/projects/4d38869141a3496c8882848ecafbe34d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:11.826196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d38869141a3496c8882848ecafbe34d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:11.828788 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:11.836127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9d33f9-a9cb-44a9-b983-a2ca792ab04b None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:11.844346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d66830-08b0-4ac6-9854-b4c1524f88a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:11.845225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d66830-08b0-4ac6-9854-b4c1524f88a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:11.845712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d66830-08b0-4ac6-9854-b4c1524f88a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:11.846151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d66830-08b0-4ac6-9854-b4c1524f88a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:11.846955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d66830-08b0-4ac6-9854-b4c1524f88a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:11.847396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d66830-08b0-4ac6-9854-b4c1524f88a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:11.849826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d66830-08b0-4ac6-9854-b4c1524f88a4 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:11.861972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 918/1850] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:11.869042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d66830-08b0-4ac6-9854-b4c1524f88a4 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:11.880107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d66830-08b0-4ac6-9854-b4c1524f88a4 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:11.899827 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d38869141a3496c8882848ecafbe34d', 'request_id': 'req-9750ddec-1909-479b-9e95-691b4cf1030a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.921950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d38869141a3496c8882848ecafbe34d', 'request_id': 'req-9750ddec-1909-479b-9e95-691b4cf1030a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.931688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d38869141a3496c8882848ecafbe34d', 'request_id': 'req-9750ddec-1909-479b-9e95-691b4cf1030a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.937532 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:11.952632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] The token cache is being invalidate because project 4d38869141a3496c8882848ecafbe34d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:11.953447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.954529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:11.958146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 933/1851] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:11 2022] DELETE /identity/v3/projects/4d38869141a3496c8882848ecafbe34d => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:11.959050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:11.961390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:11.962962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667d5861-6834-41ad-a082-ce1ab6bf346e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:11.964001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667d5861-6834-41ad-a082-ce1ab6bf346e admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:11.964166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667d5861-6834-41ad-a082-ce1ab6bf346e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:11.964441 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667d5861-6834-41ad-a082-ce1ab6bf346e admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:11.966233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667d5861-6834-41ad-a082-ce1ab6bf346e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:11.974639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9750ddec-1909-479b-9e95-691b4cf1030a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:11.976992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667d5861-6834-41ad-a082-ce1ab6bf346e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:11.981544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:11.982217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:11.982542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:11.983047 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] PATH_INFO: `/v3/projects/9925b233462a4323809ececfa452ded7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:11.983939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9925b233462a4323809ececfa452ded7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:11.985928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.007442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 919/1852] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:11 2022] POST /identity/v3/projects => generated 399 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:12.009042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a340ad6-513b-4611-a3bc-d4239bb22aaa None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.009327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a340ad6-513b-4611-a3bc-d4239bb22aaa None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.009598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a340ad6-513b-4611-a3bc-d4239bb22aaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.036854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8a340ad6-513b-4611-a3bc-d4239bb22aaa None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:12.038511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8a340ad6-513b-4611-a3bc-d4239bb22aaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:12.051871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9925b233462a4323809ececfa452ded7', 'request_id': 'req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:12.061832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9925b233462a4323809ececfa452ded7', 'request_id': 'req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:12.070443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9925b233462a4323809ececfa452ded7', 'request_id': 'req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:12.083269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] The token cache is being invalidate because project 9925b233462a4323809ececfa452ded7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:12.083648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:12.084393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:12.089756 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 934/1853] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:11 2022] DELETE /identity/v3/projects/9925b233462a4323809ececfa452ded7 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:12.089756 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.089756 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:12.129658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 920/1854] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:12.131522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a125f-7532-4227-b5c0-44f91fb2f635 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.132104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a125f-7532-4227-b5c0-44f91fb2f635 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.132695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a125f-7532-4227-b5c0-44f91fb2f635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.166337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b60adc6-8eb2-4dce-b247-e0b1fd8a3150 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.167419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-139a125f-7532-4227-b5c0-44f91fb2f635 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:12.168687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-139a125f-7532-4227-b5c0-44f91fb2f635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:12.170973 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f62883-765e-4795-a643-1b3194ad2df4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.171520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f62883-765e-4795-a643-1b3194ad2df4 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.171793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f62883-765e-4795-a643-1b3194ad2df4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.172069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f62883-765e-4795-a643-1b3194ad2df4 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.172689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f62883-765e-4795-a643-1b3194ad2df4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.174679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f62883-765e-4795-a643-1b3194ad2df4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.177022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 921/1855] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:12.178225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00431911-5ae3-43eb-8b81-af5cf90855e6 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.178504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00431911-5ae3-43eb-8b81-af5cf90855e6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.178771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00431911-5ae3-43eb-8b81-af5cf90855e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.186668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=81,keystone:DELETE=14,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:12.204547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 935/1856] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/users => generated 300 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:12.204733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-00431911-5ae3-43eb-8b81-af5cf90855e6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:12.204828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316b2d22-629a-477d-a129-3bb41006f269 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.205337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-00431911-5ae3-43eb-8b81-af5cf90855e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:12.205508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316b2d22-629a-477d-a129-3bb41006f269 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.205508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316b2d22-629a-477d-a129-3bb41006f269 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.212429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 922/1857] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:12.213229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00431911-5ae3-43eb-8b81-af5cf90855e6 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.224623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00431911-5ae3-43eb-8b81-af5cf90855e6 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.224962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-316b2d22-629a-477d-a129-3bb41006f269 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:12.226201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-316b2d22-629a-477d-a129-3bb41006f269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:12.229371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c77f28-0843-489a-8e76-6c06a275a4d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.230038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c77f28-0843-489a-8e76-6c06a275a4d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.230287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c77f28-0843-489a-8e76-6c06a275a4d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.230561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c77f28-0843-489a-8e76-6c06a275a4d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.231264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c77f28-0843-489a-8e76-6c06a275a4d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.232880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c77f28-0843-489a-8e76-6c06a275a4d1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.233999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 936/1858] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:12.234824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316b2d22-629a-477d-a129-3bb41006f269 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.244111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316b2d22-629a-477d-a129-3bb41006f269 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.246734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 923/1859] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:12 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:12.248902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798db5b0-45c4-49f8-88a1-bdf3911aab1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.249576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798db5b0-45c4-49f8-88a1-bdf3911aab1c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.249760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798db5b0-45c4-49f8-88a1-bdf3911aab1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.250030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798db5b0-45c4-49f8-88a1-bdf3911aab1c admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.250350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c77f28-0843-489a-8e76-6c06a275a4d1 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.250692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798db5b0-45c4-49f8-88a1-bdf3911aab1c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:12.250995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798db5b0-45c4-49f8-88a1-bdf3911aab1c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.252582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798db5b0-45c4-49f8-88a1-bdf3911aab1c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.259715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c77f28-0843-489a-8e76-6c06a275a4d1 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.264213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2920ffd-3536-454e-acfc-448b6d703f0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.264334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 937/1860] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:12.264796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2920ffd-3536-454e-acfc-448b6d703f0b admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.265077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2920ffd-3536-454e-acfc-448b6d703f0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.265166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798db5b0-45c4-49f8-88a1-bdf3911aab1c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.265346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2920ffd-3536-454e-acfc-448b6d703f0b admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.265977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2920ffd-3536-454e-acfc-448b6d703f0b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:12.266385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2920ffd-3536-454e-acfc-448b6d703f0b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.267956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2920ffd-3536-454e-acfc-448b6d703f0b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.275057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798db5b0-45c4-49f8-88a1-bdf3911aab1c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.279721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a080e453-96b0-4606-b4ba-f0e1e320dd8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.280310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a080e453-96b0-4606-b4ba-f0e1e320dd8e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.280583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a080e453-96b0-4606-b4ba-f0e1e320dd8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.280855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a080e453-96b0-4606-b4ba-f0e1e320dd8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.281483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a080e453-96b0-4606-b4ba-f0e1e320dd8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.283161 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a080e453-96b0-4606-b4ba-f0e1e320dd8e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.284129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 924/1861] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:12.284979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2920ffd-3536-454e-acfc-448b6d703f0b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.297963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 938/1862] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:12.297963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a080e453-96b0-4606-b4ba-f0e1e320dd8e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.298395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2920ffd-3536-454e-acfc-448b6d703f0b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.301691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47447e42-e476-4295-bf9b-d11e696af886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.302442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47447e42-e476-4295-bf9b-d11e696af886 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.302720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47447e42-e476-4295-bf9b-d11e696af886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.302992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47447e42-e476-4295-bf9b-d11e696af886 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.303804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47447e42-e476-4295-bf9b-d11e696af886 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.305712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47447e42-e476-4295-bf9b-d11e696af886 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.309864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a080e453-96b0-4606-b4ba-f0e1e320dd8e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.315234 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637c71a4-b742-4d7b-8d9d-0414a2b7b160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.315905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637c71a4-b742-4d7b-8d9d-0414a2b7b160 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.316254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637c71a4-b742-4d7b-8d9d-0414a2b7b160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.316595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637c71a4-b742-4d7b-8d9d-0414a2b7b160 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.317320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637c71a4-b742-4d7b-8d9d-0414a2b7b160 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.319952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637c71a4-b742-4d7b-8d9d-0414a2b7b160 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.336042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 925/1863] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:12.336928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47447e42-e476-4295-bf9b-d11e696af886 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.353991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 939/1864] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/projects => generated 387 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:12.353991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637c71a4-b742-4d7b-8d9d-0414a2b7b160 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.412927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47447e42-e476-4295-bf9b-d11e696af886 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.417752 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7735681a-d76d-4cdc-ac74-b5ac90219fc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.418949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7735681a-d76d-4cdc-ac74-b5ac90219fc9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.418949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7735681a-d76d-4cdc-ac74-b5ac90219fc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.418949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7735681a-d76d-4cdc-ac74-b5ac90219fc9 admin admin] PATH_INFO: `/v3/projects/f312db0eb0094badbcdd306789b776d2/users/c213b79123bb4fd19018aa1056bc4c66/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.420276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637c71a4-b742-4d7b-8d9d-0414a2b7b160 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.425097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6093fcb5-b90e-43d1-bc4a-5e781496c1bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.425636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6093fcb5-b90e-43d1-bc4a-5e781496c1bd admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.425912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6093fcb5-b90e-43d1-bc4a-5e781496c1bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.426240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6093fcb5-b90e-43d1-bc4a-5e781496c1bd admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.426869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6093fcb5-b90e-43d1-bc4a-5e781496c1bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.428792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6093fcb5-b90e-43d1-bc4a-5e781496c1bd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.436451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7735681a-d76d-4cdc-ac74-b5ac90219fc9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f312db0eb0094badbcdd306789b776d2, user_id=c213b79123bb4fd19018aa1056bc4c66, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.439635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7735681a-d76d-4cdc-ac74-b5ac90219fc9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.443176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:12.451722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 940/1865] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/users => generated 287 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:12.452490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6093fcb5-b90e-43d1-bc4a-5e781496c1bd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.457461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 926/1866] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:12 2022] PUT /identity/v3/projects/f312db0eb0094badbcdd306789b776d2/users/c213b79123bb4fd19018aa1056bc4c66/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:12.463073 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7735681a-d76d-4cdc-ac74-b5ac90219fc9 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.476097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6093fcb5-b90e-43d1-bc4a-5e781496c1bd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.534726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dbae3d-2e22-4e58-b507-fb0df0d86951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.535330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dbae3d-2e22-4e58-b507-fb0df0d86951 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.536130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dbae3d-2e22-4e58-b507-fb0df0d86951 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.536409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dbae3d-2e22-4e58-b507-fb0df0d86951 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.536691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7735681a-d76d-4cdc-ac74-b5ac90219fc9 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.537030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41dbae3d-2e22-4e58-b507-fb0df0d86951 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.538969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41dbae3d-2e22-4e58-b507-fb0df0d86951 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.541175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb927f72-5276-41d2-a19f-6ccd1f14750a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.541780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb927f72-5276-41d2-a19f-6ccd1f14750a admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.542068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb927f72-5276-41d2-a19f-6ccd1f14750a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.542357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb927f72-5276-41d2-a19f-6ccd1f14750a admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.542993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb927f72-5276-41d2-a19f-6ccd1f14750a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.544587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb927f72-5276-41d2-a19f-6ccd1f14750a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.554791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 927/1867] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:12 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:12.555809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b184555-52cd-4c26-b250-5efce17f927d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.555933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b184555-52cd-4c26-b250-5efce17f927d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.556197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b184555-52cd-4c26-b250-5efce17f927d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.561117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 941/1868] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/users => generated 294 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:12.562004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dbae3d-2e22-4e58-b507-fb0df0d86951 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.573108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dbae3d-2e22-4e58-b507-fb0df0d86951 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.577941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79f83bf-3954-4f32-95c3-4f09d6f98f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.578552 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79f83bf-3954-4f32-95c3-4f09d6f98f2f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.578823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79f83bf-3954-4f32-95c3-4f09d6f98f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.579089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79f83bf-3954-4f32-95c3-4f09d6f98f2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.579708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79f83bf-3954-4f32-95c3-4f09d6f98f2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.581272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79f83bf-3954-4f32-95c3-4f09d6f98f2f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.584074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8b184555-52cd-4c26-b250-5efce17f927d None None] MFA Rules not processed for user `c213b79123bb4fd19018aa1056bc4c66`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:12.585268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8b184555-52cd-4c26-b250-5efce17f927d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:12.591959 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 942/1869] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:12.592739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79f83bf-3954-4f32-95c3-4f09d6f98f2f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.604019 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79f83bf-3954-4f32-95c3-4f09d6f98f2f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.608477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f8b7e7-16c5-464f-bdf8-02821a4e1bbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.609073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f8b7e7-16c5-464f-bdf8-02821a4e1bbb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.609349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f8b7e7-16c5-464f-bdf8-02821a4e1bbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.609620 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f8b7e7-16c5-464f-bdf8-02821a4e1bbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.610279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f8b7e7-16c5-464f-bdf8-02821a4e1bbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.611842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f8b7e7-16c5-464f-bdf8-02821a4e1bbb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.616881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:12.624093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 943/1870] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:12.624882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f8b7e7-16c5-464f-bdf8-02821a4e1bbb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.637095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f8b7e7-16c5-464f-bdf8-02821a4e1bbb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.642083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764d0a5f-6e65-4cf3-ba46-59f69a368f48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.642709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764d0a5f-6e65-4cf3-ba46-59f69a368f48 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.643088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764d0a5f-6e65-4cf3-ba46-59f69a368f48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.643361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764d0a5f-6e65-4cf3-ba46-59f69a368f48 admin admin] PATH_INFO: `/v3/projects/dd739ef48e834bdeba7d44089e323879/users/d21b0315b5c54467b0da2667ac1570b9/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.653574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 928/1871] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:12.654506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b184555-52cd-4c26-b250-5efce17f927d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.660946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764d0a5f-6e65-4cf3-ba46-59f69a368f48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd739ef48e834bdeba7d44089e323879, user_id=d21b0315b5c54467b0da2667ac1570b9, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.663649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764d0a5f-6e65-4cf3-ba46-59f69a368f48 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.664755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b184555-52cd-4c26-b250-5efce17f927d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.669913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf18cf9-d406-4b6c-bde1-31cc00cbb1d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.670561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf18cf9-d406-4b6c-bde1-31cc00cbb1d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.670957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf18cf9-d406-4b6c-bde1-31cc00cbb1d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.671342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf18cf9-d406-4b6c-bde1-31cc00cbb1d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.671974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf18cf9-d406-4b6c-bde1-31cc00cbb1d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.673802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf18cf9-d406-4b6c-bde1-31cc00cbb1d1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.678423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 944/1872] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:12 2022] PUT /identity/v3/projects/dd739ef48e834bdeba7d44089e323879/users/d21b0315b5c54467b0da2667ac1570b9/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:12.679814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764d0a5f-6e65-4cf3-ba46-59f69a368f48 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.684298 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 929/1873] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:12.690200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf18cf9-d406-4b6c-bde1-31cc00cbb1d1 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.700542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf18cf9-d406-4b6c-bde1-31cc00cbb1d1 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.742935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764d0a5f-6e65-4cf3-ba46-59f69a368f48 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.748080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe533e8-e226-4bd9-98de-82d8968681a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.749069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe533e8-e226-4bd9-98de-82d8968681a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.749340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe533e8-e226-4bd9-98de-82d8968681a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.749609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe533e8-e226-4bd9-98de-82d8968681a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.750287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe533e8-e226-4bd9-98de-82d8968681a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.752213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe533e8-e226-4bd9-98de-82d8968681a2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.761022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ebd43-c8c1-448f-afad-3aa48b55bdb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.761713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ebd43-c8c1-448f-afad-3aa48b55bdb3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.761992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ebd43-c8c1-448f-afad-3aa48b55bdb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.762767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ebd43-c8c1-448f-afad-3aa48b55bdb3 admin admin] PATH_INFO: `/v3/projects/84d26c9884eb46b9967b2f34bf1f212a/users/e2209ce29c0e4edd8969c1521061ad2b/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.777564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:12.781625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ebd43-c8c1-448f-afad-3aa48b55bdb3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=84d26c9884eb46b9967b2f34bf1f212a, user_id=e2209ce29c0e4edd8969c1521061ad2b, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.781779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 945/1874] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/projects => generated 401 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:12.783222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd9616-bce0-42ba-9700-de2c6da11c53 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.783499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd9616-bce0-42ba-9700-de2c6da11c53 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.783599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ebd43-c8c1-448f-afad-3aa48b55bdb3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.783769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfd9616-bce0-42ba-9700-de2c6da11c53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.798893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 930/1875] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:12 2022] PUT /identity/v3/projects/84d26c9884eb46b9967b2f34bf1f212a/users/e2209ce29c0e4edd8969c1521061ad2b/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:12.799689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ebd43-c8c1-448f-afad-3aa48b55bdb3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.830757 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9cfd9616-bce0-42ba-9700-de2c6da11c53 None None] MFA Rules not processed for user `d21b0315b5c54467b0da2667ac1570b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:12.832450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9cfd9616-bce0-42ba-9700-de2c6da11c53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:12.878477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ebd43-c8c1-448f-afad-3aa48b55bdb3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.883877 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaef0d5-febe-4bc1-ab99-310ef85a2247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.884484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaef0d5-febe-4bc1-ab99-310ef85a2247 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.884761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaef0d5-febe-4bc1-ab99-310ef85a2247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.885034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcaef0d5-febe-4bc1-ab99-310ef85a2247 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.885666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcaef0d5-febe-4bc1-ab99-310ef85a2247 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:12.888096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcaef0d5-febe-4bc1-ab99-310ef85a2247 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:12.909223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 946/1876] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:12.909223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33629c78-9622-42a3-8da2-a6ee17ca339a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.909223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33629c78-9622-42a3-8da2-a6ee17ca339a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.909223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33629c78-9622-42a3-8da2-a6ee17ca339a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:12.922150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 931/1877] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/users => generated 307 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:12.922971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaef0d5-febe-4bc1-ab99-310ef85a2247 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:12.927050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-bcaef0d5-febe-4bc1-ab99-310ef85a2247 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:12.947382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-33629c78-9622-42a3-8da2-a6ee17ca339a None None] MFA Rules not processed for user `e2209ce29c0e4edd8969c1521061ad2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:12.948632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-33629c78-9622-42a3-8da2-a6ee17ca339a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:12.985913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcaef0d5-febe-4bc1-ab99-310ef85a2247 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:12.987887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:12.990788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0137a7-c5a4-49a7-8612-7b89dfc09c93 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:12.991481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0137a7-c5a4-49a7-8612-7b89dfc09c93 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:12.991834 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0137a7-c5a4-49a7-8612-7b89dfc09c93 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:12.992259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0137a7-c5a4-49a7-8612-7b89dfc09c93 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.001212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0137a7-c5a4-49a7-8612-7b89dfc09c93 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.002761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0137a7-c5a4-49a7-8612-7b89dfc09c93 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.013073 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 932/1878] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:12 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:13.013896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0137a7-c5a4-49a7-8612-7b89dfc09c93 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.019657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 947/1879] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:12 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:13.024254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33629c78-9622-42a3-8da2-a6ee17ca339a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.024835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0137a7-c5a4-49a7-8612-7b89dfc09c93 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.031570 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658809a8-b00d-40a3-a36c-3e524cc96c0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.032187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658809a8-b00d-40a3-a36c-3e524cc96c0d admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.032516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658809a8-b00d-40a3-a36c-3e524cc96c0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.032787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658809a8-b00d-40a3-a36c-3e524cc96c0d admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.033431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658809a8-b00d-40a3-a36c-3e524cc96c0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.035096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658809a8-b00d-40a3-a36c-3e524cc96c0d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.041953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33629c78-9622-42a3-8da2-a6ee17ca339a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.047249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16a58d0-cc00-470f-82b9-55a0ff3c7c94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.047618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 933/1880] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:13 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:13.047854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16a58d0-cc00-470f-82b9-55a0ff3c7c94 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.048125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16a58d0-cc00-470f-82b9-55a0ff3c7c94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.048391 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16a58d0-cc00-470f-82b9-55a0ff3c7c94 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.049013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16a58d0-cc00-470f-82b9-55a0ff3c7c94 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.051544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16a58d0-cc00-470f-82b9-55a0ff3c7c94 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.051900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658809a8-b00d-40a3-a36c-3e524cc96c0d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.062214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658809a8-b00d-40a3-a36c-3e524cc96c0d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.067427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f21432-cea5-450f-a358-0586d7f7243d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.068077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f21432-cea5-450f-a358-0586d7f7243d admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.068353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f21432-cea5-450f-a358-0586d7f7243d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.068633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f21432-cea5-450f-a358-0586d7f7243d admin admin] PATH_INFO: `/v3/projects/f7ff3986cae84b2f92a36f86c384411e/users/33894bc40e3f4deeb549e7293b456b1b/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.080534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 948/1881] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:13 2022] POST /identity/v3/projects => generated 387 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:13.084699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16a58d0-cc00-470f-82b9-55a0ff3c7c94 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.086738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f21432-cea5-450f-a358-0586d7f7243d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7ff3986cae84b2f92a36f86c384411e, user_id=33894bc40e3f4deeb549e7293b456b1b, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.089387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f21432-cea5-450f-a358-0586d7f7243d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.105481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 934/1882] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:13 2022] PUT /identity/v3/projects/f7ff3986cae84b2f92a36f86c384411e/users/33894bc40e3f4deeb549e7293b456b1b/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:13.109587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f21432-cea5-450f-a358-0586d7f7243d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.171709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16a58d0-cc00-470f-82b9-55a0ff3c7c94 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.178330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:13.197336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f21432-cea5-450f-a358-0586d7f7243d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.202499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a600adfd-1d8d-4664-93e0-77d2e2b8ccc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.203370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a600adfd-1d8d-4664-93e0-77d2e2b8ccc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.203813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a600adfd-1d8d-4664-93e0-77d2e2b8ccc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.204303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a600adfd-1d8d-4664-93e0-77d2e2b8ccc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.205231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a600adfd-1d8d-4664-93e0-77d2e2b8ccc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.207743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a600adfd-1d8d-4664-93e0-77d2e2b8ccc3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.231411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 935/1883] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:13 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:13.236261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a600adfd-1d8d-4664-93e0-77d2e2b8ccc3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.251405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82802196-b821-482a-a313-6c54ff384033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.251405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82802196-b821-482a-a313-6c54ff384033 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.251992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82802196-b821-482a-a313-6c54ff384033 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.251992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82802196-b821-482a-a313-6c54ff384033 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.252481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82802196-b821-482a-a313-6c54ff384033 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.254954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82802196-b821-482a-a313-6c54ff384033 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.261713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a600adfd-1d8d-4664-93e0-77d2e2b8ccc3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.269253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b65a41-f288-4ee4-9bfe-970f50d9108c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.270191 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b65a41-f288-4ee4-9bfe-970f50d9108c admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.270437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b65a41-f288-4ee4-9bfe-970f50d9108c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.270853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b65a41-f288-4ee4-9bfe-970f50d9108c admin admin] PATH_INFO: `/v3/projects/f7ff3986cae84b2f92a36f86c384411e/users/33894bc40e3f4deeb549e7293b456b1b/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.272949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b65a41-f288-4ee4-9bfe-970f50d9108c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7ff3986cae84b2f92a36f86c384411e, user_id=33894bc40e3f4deeb549e7293b456b1b, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.275725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b65a41-f288-4ee4-9bfe-970f50d9108c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.287396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 949/1884] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:13 2022] POST /identity/v3/users => generated 294 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:13.290414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 936/1885] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:13 2022] PUT /identity/v3/projects/f7ff3986cae84b2f92a36f86c384411e/users/33894bc40e3f4deeb549e7293b456b1b/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:13.291309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b65a41-f288-4ee4-9bfe-970f50d9108c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.300102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1248ccce-33ee-45e6-9c93-34e83ad6acda None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.300745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1248ccce-33ee-45e6-9c93-34e83ad6acda None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.301557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1248ccce-33ee-45e6-9c93-34e83ad6acda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.341897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1248ccce-33ee-45e6-9c93-34e83ad6acda None None] MFA Rules not processed for user `33894bc40e3f4deeb549e7293b456b1b`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:13.344237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1248ccce-33ee-45e6-9c93-34e83ad6acda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:13.392742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b65a41-f288-4ee4-9bfe-970f50d9108c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.405049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff859d6-4b9c-4d88-8a33-01763dcf8e6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.406207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff859d6-4b9c-4d88-8a33-01763dcf8e6b admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.406787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:13.407079 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff859d6-4b9c-4d88-8a33-01763dcf8e6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.407550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff859d6-4b9c-4d88-8a33-01763dcf8e6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.409226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff859d6-4b9c-4d88-8a33-01763dcf8e6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.411420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff859d6-4b9c-4d88-8a33-01763dcf8e6b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.426545 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 937/1886] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:13 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:13.430108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff859d6-4b9c-4d88-8a33-01763dcf8e6b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.440364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff859d6-4b9c-4d88-8a33-01763dcf8e6b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.447853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e7aeca-949f-4733-b82f-79fd406a91c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.448593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e7aeca-949f-4733-b82f-79fd406a91c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.449043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e7aeca-949f-4733-b82f-79fd406a91c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.449408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e7aeca-949f-4733-b82f-79fd406a91c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.450106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e7aeca-949f-4733-b82f-79fd406a91c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.452072 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e7aeca-949f-4733-b82f-79fd406a91c7 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.453223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 950/1887] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:13 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:13.462039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1248ccce-33ee-45e6-9c93-34e83ad6acda None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.466130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 938/1888] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:13 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:13.469761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e7aeca-949f-4733-b82f-79fd406a91c7 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.479907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e7aeca-949f-4733-b82f-79fd406a91c7 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.484740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2c8704-b73c-49e1-aa4e-5f5c95310720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.485389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2c8704-b73c-49e1-aa4e-5f5c95310720 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.485685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2c8704-b73c-49e1-aa4e-5f5c95310720 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.485955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2c8704-b73c-49e1-aa4e-5f5c95310720 admin admin] PATH_INFO: `/v3/projects/b7c1d2e23f1c47fea1b528fb9365d583/users/7f8238108f4e4b8f98b2c4d6c901184c/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.501275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2c8704-b73c-49e1-aa4e-5f5c95310720 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7c1d2e23f1c47fea1b528fb9365d583, user_id=7f8238108f4e4b8f98b2c4d6c901184c, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.503975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2c8704-b73c-49e1-aa4e-5f5c95310720 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.518545 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 939/1889] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:13 2022] PUT /identity/v3/projects/b7c1d2e23f1c47fea1b528fb9365d583/users/7f8238108f4e4b8f98b2c4d6c901184c/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:13.528597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1248ccce-33ee-45e6-9c93-34e83ad6acda None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.529829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fb1712-510b-4b25-aa79-4636ec8572b0 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.529941 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fb1712-510b-4b25-aa79-4636ec8572b0 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.530303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fb1712-510b-4b25-aa79-4636ec8572b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.560324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e0fb1712-510b-4b25-aa79-4636ec8572b0 None None] MFA Rules not processed for user `7f8238108f4e4b8f98b2c4d6c901184c`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:13.561635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e0fb1712-510b-4b25-aa79-4636ec8572b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:13.588203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1191f803-6b96-4f2a-bc25-e6e0b05f076b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.588809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1191f803-6b96-4f2a-bc25-e6e0b05f076b service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.589082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1191f803-6b96-4f2a-bc25-e6e0b05f076b service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.589447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1191f803-6b96-4f2a-bc25-e6e0b05f076b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.641560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 940/1890] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:13 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:13.645777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fb1712-510b-4b25-aa79-4636ec8572b0 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.652998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1191f803-6b96-4f2a-bc25-e6e0b05f076b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.654798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1191f803-6b96-4f2a-bc25-e6e0b05f076b service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.666575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 951/1891] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:13 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:13.670461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:13.711473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fb1712-510b-4b25-aa79-4636ec8572b0 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.715980 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e20427-a8b0-4366-99a7-a17d3ede5b10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.716581 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e20427-a8b0-4366-99a7-a17d3ede5b10 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.716857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e20427-a8b0-4366-99a7-a17d3ede5b10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.717129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e20427-a8b0-4366-99a7-a17d3ede5b10 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.717756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e20427-a8b0-4366-99a7-a17d3ede5b10 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.719644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e20427-a8b0-4366-99a7-a17d3ede5b10 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.746929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1191f803-6b96-4f2a-bc25-e6e0b05f076b service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.749913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 941/1892] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:13 2022] POST /identity/v3/projects => generated 385 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:13.753741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e20427-a8b0-4366-99a7-a17d3ede5b10 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.755899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1191f803-6b96-4f2a-bc25-e6e0b05f076b service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.797597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:13.807660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e004d4b6-ac5d-4db7-a447-149799461fe5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.808545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e004d4b6-ac5d-4db7-a447-149799461fe5 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.808901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e004d4b6-ac5d-4db7-a447-149799461fe5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.809180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e004d4b6-ac5d-4db7-a447-149799461fe5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.818373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e20427-a8b0-4366-99a7-a17d3ede5b10 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.823292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733f5e57-af95-44cc-8f41-8bf523eb740f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.823982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f5e57-af95-44cc-8f41-8bf523eb740f admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.824405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f5e57-af95-44cc-8f41-8bf523eb740f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.824684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f5e57-af95-44cc-8f41-8bf523eb740f admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.825433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733f5e57-af95-44cc-8f41-8bf523eb740f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.827416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733f5e57-af95-44cc-8f41-8bf523eb740f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.856645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 942/1893] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:13 2022] POST /identity/v3/users => generated 299 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:13.857629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733f5e57-af95-44cc-8f41-8bf523eb740f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.873437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e004d4b6-ac5d-4db7-a447-149799461fe5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.874882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e004d4b6-ac5d-4db7-a447-149799461fe5 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.877684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733f5e57-af95-44cc-8f41-8bf523eb740f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.883628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2911a5-b66c-4608-ae23-366b254efde3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.884235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2911a5-b66c-4608-ae23-366b254efde3 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.884508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 952/1894] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:13 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:13.884635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2911a5-b66c-4608-ae23-366b254efde3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.884830 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2911a5-b66c-4608-ae23-366b254efde3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.885573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2911a5-b66c-4608-ae23-366b254efde3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.887169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2911a5-b66c-4608-ae23-366b254efde3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.897854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 943/1895] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:13 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:13.902048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e004d4b6-ac5d-4db7-a447-149799461fe5 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.913379 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e004d4b6-ac5d-4db7-a447-149799461fe5 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:13.918076 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94e1b4d-bffd-4edb-8a9a-4b2c57b641ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:13.918722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94e1b4d-bffd-4edb-8a9a-4b2c57b641ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:13.919027 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94e1b4d-bffd-4edb-8a9a-4b2c57b641ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:13.919297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94e1b4d-bffd-4edb-8a9a-4b2c57b641ce admin admin] PATH_INFO: `/v3/projects/4abd4fec37f64eaea237212dbf23783a/users/f9d9424e9f014ef69c70bb7275ffefb1/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:13.935928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94e1b4d-bffd-4edb-8a9a-4b2c57b641ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=4abd4fec37f64eaea237212dbf23783a, user_id=f9d9424e9f014ef69c70bb7275ffefb1, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:13.939009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94e1b4d-bffd-4edb-8a9a-4b2c57b641ce admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:13.957770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 953/1896] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:13 2022] PUT /identity/v3/projects/4abd4fec37f64eaea237212dbf23783a/users/f9d9424e9f014ef69c70bb7275ffefb1/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:13.962593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2911a5-b66c-4608-ae23-366b254efde3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:13.973772 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2911a5-b66c-4608-ae23-366b254efde3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:14.047366 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1aecc16-9db9-4c49-9192-0b2d746a48b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:14.048573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aecc16-9db9-4c49-9192-0b2d746a48b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:14.049204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aecc16-9db9-4c49-9192-0b2d746a48b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:14.049514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1aecc16-9db9-4c49-9192-0b2d746a48b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:14.050287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1aecc16-9db9-4c49-9192-0b2d746a48b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:14.052605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1aecc16-9db9-4c49-9192-0b2d746a48b2 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:14.063627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 944/1897] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:13 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:14.067469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94e1b4d-bffd-4edb-8a9a-4b2c57b641ce admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:14.078273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94e1b4d-bffd-4edb-8a9a-4b2c57b641ce admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:14.083870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29505c62-76fe-42de-91bf-3089f6413998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:14.084929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29505c62-76fe-42de-91bf-3089f6413998 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:14.084929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29505c62-76fe-42de-91bf-3089f6413998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:14.085128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29505c62-76fe-42de-91bf-3089f6413998 admin admin] PATH_INFO: `/v3/projects/4abd4fec37f64eaea237212dbf23783a/users/f9d9424e9f014ef69c70bb7275ffefb1/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:14.087600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29505c62-76fe-42de-91bf-3089f6413998 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4abd4fec37f64eaea237212dbf23783a, user_id=f9d9424e9f014ef69c70bb7275ffefb1, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:14.089655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29505c62-76fe-42de-91bf-3089f6413998 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:14.103785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 954/1898] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:14 2022] PUT /identity/v3/projects/4abd4fec37f64eaea237212dbf23783a/users/f9d9424e9f014ef69c70bb7275ffefb1/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:14.115708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012ed49c-e8e8-4f00-98c4-7623b542a9c9 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:14.116102 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012ed49c-e8e8-4f00-98c4-7623b542a9c9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:14.116822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012ed49c-e8e8-4f00-98c4-7623b542a9c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:14.152146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-012ed49c-e8e8-4f00-98c4-7623b542a9c9 None None] MFA Rules not processed for user `f9d9424e9f014ef69c70bb7275ffefb1`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:14.153479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-012ed49c-e8e8-4f00-98c4-7623b542a9c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:14.164319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:14.235938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 945/1899] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:14 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:14.247835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29505c62-76fe-42de-91bf-3089f6413998 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:14.312538 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29505c62-76fe-42de-91bf-3089f6413998 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:14.316165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139a6bd1-c30f-40d4-8383-2b726c3df1f4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:14.316765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a6bd1-c30f-40d4-8383-2b726c3df1f4 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:14.317039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a6bd1-c30f-40d4-8383-2b726c3df1f4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:14.317308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a6bd1-c30f-40d4-8383-2b726c3df1f4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:14.326207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139a6bd1-c30f-40d4-8383-2b726c3df1f4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:14.327632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139a6bd1-c30f-40d4-8383-2b726c3df1f4 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:14.337346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 955/1900] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:14 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:14.920459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012ed49c-e8e8-4f00-98c4-7623b542a9c9 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:14.921960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-012ed49c-e8e8-4f00-98c4-7623b542a9c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:14.933739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012ed49c-e8e8-4f00-98c4-7623b542a9c9 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:14.937305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0aefade-286a-44cd-8815-c993716ebf42 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:14.937936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0aefade-286a-44cd-8815-c993716ebf42 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:14.938475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0aefade-286a-44cd-8815-c993716ebf42 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:14.938562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0aefade-286a-44cd-8815-c993716ebf42 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:14.994429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0aefade-286a-44cd-8815-c993716ebf42 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:14.995850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0aefade-286a-44cd-8815-c993716ebf42 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:15.004955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 946/1901] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:14 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:17.886550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f060c-cba8-41d1-b9ed-af77683d628a None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:17.887075 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f060c-cba8-41d1-b9ed-af77683d628a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:17.887449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f060c-cba8-41d1-b9ed-af77683d628a None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:17.888629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 956/1902] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:55:17 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:55:17.911128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a2b67e-b481-47d7-8a9c-caf9dd986101 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:17.911681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a2b67e-b481-47d7-8a9c-caf9dd986101 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:17.911681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a2b67e-b481-47d7-8a9c-caf9dd986101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:17.983160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-b3a2b67e-b481-47d7-8a9c-caf9dd986101 None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:17.984419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b3a2b67e-b481-47d7-8a9c-caf9dd986101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:18.039085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f060c-cba8-41d1-b9ed-af77683d628a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:18.066873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f060c-cba8-41d1-b9ed-af77683d628a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:18.078989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1a256f-c7b6-44a0-b585-e4862bbefffc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:18.079700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1a256f-c7b6-44a0-b585-e4862bbefffc service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:18.080007 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1a256f-c7b6-44a0-b585-e4862bbefffc service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:18.080325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1a256f-c7b6-44a0-b585-e4862bbefffc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:18.087669 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6f1a256f-c7b6-44a0-b585-e4862bbefffc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:18.216842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 947/1903] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:55:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:18.255269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1a256f-c7b6-44a0-b585-e4862bbefffc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:18.257005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1a256f-c7b6-44a0-b585-e4862bbefffc service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:18.269384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:18.282869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 957/1904] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:18 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:18.883482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a2b67e-b481-47d7-8a9c-caf9dd986101 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:18.884672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b3a2b67e-b481-47d7-8a9c-caf9dd986101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:18.902027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a2b67e-b481-47d7-8a9c-caf9dd986101 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:18.917721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970fcdcd-c0a1-4ae4-b316-22c5ae83189f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:18.919513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970fcdcd-c0a1-4ae4-b316-22c5ae83189f service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:18.920175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970fcdcd-c0a1-4ae4-b316-22c5ae83189f service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:18.920175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970fcdcd-c0a1-4ae4-b316-22c5ae83189f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:18.936133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970fcdcd-c0a1-4ae4-b316-22c5ae83189f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:18.938016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970fcdcd-c0a1-4ae4-b316-22c5ae83189f service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:18.963495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 948/1905] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:18 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:19.898077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1a256f-c7b6-44a0-b585-e4862bbefffc service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:19.965563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1a256f-c7b6-44a0-b585-e4862bbefffc service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:19.970008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:19.970651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:19.970922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:19.971186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] PATH_INFO: `/v3/users/d21b0315b5c54467b0da2667ac1570b9` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:19.972284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d21b0315b5c54467b0da2667ac1570b9)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:19.974239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:20.032408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd21b0315b5c54467b0da2667ac1570b9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:20.040437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd21b0315b5c54467b0da2667ac1570b9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:20.041385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd21b0315b5c54467b0da2667ac1570b9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:20.042234 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd21b0315b5c54467b0da2667ac1570b9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:20.052566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd21b0315b5c54467b0da2667ac1570b9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:20.093158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 958/1906] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:19 2022] DELETE /identity/v3/users/d21b0315b5c54467b0da2667ac1570b9 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:20.109991 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922fd669-d2c0-46b7-b403-e7d39a00f1b9 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:20.110523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922fd669-d2c0-46b7-b403-e7d39a00f1b9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:20.110629 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922fd669-d2c0-46b7-b403-e7d39a00f1b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:20.140577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-922fd669-d2c0-46b7-b403-e7d39a00f1b9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:20.143346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-922fd669-d2c0-46b7-b403-e7d39a00f1b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:20.167316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:20.214505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 949/1907] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:20.224062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:20.225527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:20.290015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7105c0-0bb2-4b20-a5b3-57d351c44e3a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:20.294300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4aac9a-7f77-4796-8f6e-4fb0853414ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:20.295781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4aac9a-7f77-4796-8f6e-4fb0853414ef service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:20.296144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4aac9a-7f77-4796-8f6e-4fb0853414ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:20.296510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4aac9a-7f77-4796-8f6e-4fb0853414ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:20.307524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4aac9a-7f77-4796-8f6e-4fb0853414ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:20.309156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4aac9a-7f77-4796-8f6e-4fb0853414ef service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:20.320057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 959/1908] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:20.603676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922fd669-d2c0-46b7-b403-e7d39a00f1b9 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:20.605097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-922fd669-d2c0-46b7-b403-e7d39a00f1b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:20.617481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922fd669-d2c0-46b7-b403-e7d39a00f1b9 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:20.623352 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:20.624083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:20.624363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:20.624634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] PATH_INFO: `/v3/projects/dd739ef48e834bdeba7d44089e323879` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:20.625788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd739ef48e834bdeba7d44089e323879)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:20.627759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:20.674510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd739ef48e834bdeba7d44089e323879', 'request_id': 'req-8c5a4ad2-ee28-4f61-937f-404864ca4489'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:20.681463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd739ef48e834bdeba7d44089e323879', 'request_id': 'req-8c5a4ad2-ee28-4f61-937f-404864ca4489'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:20.687519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd739ef48e834bdeba7d44089e323879', 'request_id': 'req-8c5a4ad2-ee28-4f61-937f-404864ca4489'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:20.697981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] The token cache is being invalidate because project dd739ef48e834bdeba7d44089e323879 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:20.698294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:20.698928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8c5a4ad2-ee28-4f61-937f-404864ca4489 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:20.700394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 950/1909] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:20 2022] DELETE /identity/v3/projects/dd739ef48e834bdeba7d44089e323879 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:20.705196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027a50b9-b252-46f7-ab93-0607d32dea92 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:20.705484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027a50b9-b252-46f7-ab93-0607d32dea92 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:20.705756 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027a50b9-b252-46f7-ab93-0607d32dea92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:20.725113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-027a50b9-b252-46f7-ab93-0607d32dea92 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:20.726587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-027a50b9-b252-46f7-ab93-0607d32dea92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:20.743826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:20.787933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 960/1910] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:20.795678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09049a05-ca26-451e-b254-231635a847ca None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:20.795964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09049a05-ca26-451e-b254-231635a847ca None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:20.796264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09049a05-ca26-451e-b254-231635a847ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:20.815399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-09049a05-ca26-451e-b254-231635a847ca None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:20.817091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-09049a05-ca26-451e-b254-231635a847ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:20.824751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 951/1911] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:20.828771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027a50b9-b252-46f7-ab93-0607d32dea92 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:20.839036 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027a50b9-b252-46f7-ab93-0607d32dea92 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:20.843591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5600aff5-d903-4e07-98e7-e2758929cbee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:20.844207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5600aff5-d903-4e07-98e7-e2758929cbee admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:20.844484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5600aff5-d903-4e07-98e7-e2758929cbee admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:20.844752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5600aff5-d903-4e07-98e7-e2758929cbee admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:20.845389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5600aff5-d903-4e07-98e7-e2758929cbee admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:20.845689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5600aff5-d903-4e07-98e7-e2758929cbee admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:20.847299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5600aff5-d903-4e07-98e7-e2758929cbee admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:20.858537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 961/1912] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:55:20.862776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09049a05-ca26-451e-b254-231635a847ca None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:20.872211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09049a05-ca26-451e-b254-231635a847ca None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:20.876771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93840847-a868-4358-8c76-1f7a2216de2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:20.877361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93840847-a868-4358-8c76-1f7a2216de2b admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:20.877642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93840847-a868-4358-8c76-1f7a2216de2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:20.877915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93840847-a868-4358-8c76-1f7a2216de2b admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:20.878584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93840847-a868-4358-8c76-1f7a2216de2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:20.880984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93840847-a868-4358-8c76-1f7a2216de2b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:20.912830 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 952/1913] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:20 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:20.914308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5600aff5-d903-4e07-98e7-e2758929cbee admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:20.924167 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5600aff5-d903-4e07-98e7-e2758929cbee admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:20.980333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2418b897-7e83-46ce-b57f-ee8ba2b03193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:20.980918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2418b897-7e83-46ce-b57f-ee8ba2b03193 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:20.981200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2418b897-7e83-46ce-b57f-ee8ba2b03193 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:20.981469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2418b897-7e83-46ce-b57f-ee8ba2b03193 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:20.982109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2418b897-7e83-46ce-b57f-ee8ba2b03193 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:20.984102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2418b897-7e83-46ce-b57f-ee8ba2b03193 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.007095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 962/1914] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:20 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:21.010910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93840847-a868-4358-8c76-1f7a2216de2b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.021835 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93840847-a868-4358-8c76-1f7a2216de2b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:21.027353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72c6e94-fe81-4555-908d-bdb6f50ee8f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:21.027967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72c6e94-fe81-4555-908d-bdb6f50ee8f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.028243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72c6e94-fe81-4555-908d-bdb6f50ee8f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.028658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72c6e94-fe81-4555-908d-bdb6f50ee8f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.029135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72c6e94-fe81-4555-908d-bdb6f50ee8f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:21.030751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72c6e94-fe81-4555-908d-bdb6f50ee8f9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.046183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 953/1915] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:21 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:21.047366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2418b897-7e83-46ce-b57f-ee8ba2b03193 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.057648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2418b897-7e83-46ce-b57f-ee8ba2b03193 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:21.062284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3dda5a-063a-4b0f-bf3b-ea91fa1992cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:21.062877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3dda5a-063a-4b0f-bf3b-ea91fa1992cb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.063148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3dda5a-063a-4b0f-bf3b-ea91fa1992cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.063417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3dda5a-063a-4b0f-bf3b-ea91fa1992cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.064051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3dda5a-063a-4b0f-bf3b-ea91fa1992cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:21.065635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af3dda5a-063a-4b0f-bf3b-ea91fa1992cb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.076422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 963/1916] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:21.081032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72c6e94-fe81-4555-908d-bdb6f50ee8f9 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.092234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72c6e94-fe81-4555-908d-bdb6f50ee8f9 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:21.097434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa52b16-1883-485c-a523-6c0a723561a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:21.097732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa52b16-1883-485c-a523-6c0a723561a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.098176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa52b16-1883-485c-a523-6c0a723561a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.098457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa52b16-1883-485c-a523-6c0a723561a9 admin admin] PATH_INFO: `/v3/projects/ea835a465ba545079d6b34e856015592/users/b933c13f6d10469eacd1f748bd43ac11/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.116565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa52b16-1883-485c-a523-6c0a723561a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea835a465ba545079d6b34e856015592, user_id=b933c13f6d10469eacd1f748bd43ac11, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:21.119274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa52b16-1883-485c-a523-6c0a723561a9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.145725 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 954/1917] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:21 2022] PUT /identity/v3/projects/ea835a465ba545079d6b34e856015592/users/b933c13f6d10469eacd1f748bd43ac11/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:21.147434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67489e46-c30f-4d2d-8680-3f5bba6368fa None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.147717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67489e46-c30f-4d2d-8680-3f5bba6368fa None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.147990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67489e46-c30f-4d2d-8680-3f5bba6368fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.196258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-67489e46-c30f-4d2d-8680-3f5bba6368fa None None] MFA Rules not processed for user `b933c13f6d10469eacd1f748bd43ac11`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:21.197466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-67489e46-c30f-4d2d-8680-3f5bba6368fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:21.261721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 964/1918] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:21.266369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa52b16-1883-485c-a523-6c0a723561a9 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.314235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:21.332517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa52b16-1883-485c-a523-6c0a723561a9 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:21.337196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9a7e46-02b7-41fd-8c1b-d65409ae60ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:21.337986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9a7e46-02b7-41fd-8c1b-d65409ae60ce admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.338301 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9a7e46-02b7-41fd-8c1b-d65409ae60ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.338577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9a7e46-02b7-41fd-8c1b-d65409ae60ce admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.339268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9a7e46-02b7-41fd-8c1b-d65409ae60ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:21.341345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9a7e46-02b7-41fd-8c1b-d65409ae60ce admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.368623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 955/1919] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:21 2022] POST /identity/v3/projects => generated 377 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:21.373037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67489e46-c30f-4d2d-8680-3f5bba6368fa None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.385809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67489e46-c30f-4d2d-8680-3f5bba6368fa None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:21.398363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:21.445613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424ab7c-389f-44c5-91ef-fff117166f1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:21.446248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424ab7c-389f-44c5-91ef-fff117166f1b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.446524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424ab7c-389f-44c5-91ef-fff117166f1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.447069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424ab7c-389f-44c5-91ef-fff117166f1b admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.447700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7424ab7c-389f-44c5-91ef-fff117166f1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:21.449583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7424ab7c-389f-44c5-91ef-fff117166f1b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.479221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 965/1920] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:21 2022] POST /identity/v3/users => generated 289 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:21.480423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9a7e46-02b7-41fd-8c1b-d65409ae60ce admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.491587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9a7e46-02b7-41fd-8c1b-d65409ae60ce admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:21.502920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5697b7a9-47cd-44a1-8d94-cb7d77e399bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:21.504283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5697b7a9-47cd-44a1-8d94-cb7d77e399bf admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.504763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5697b7a9-47cd-44a1-8d94-cb7d77e399bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.505133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5697b7a9-47cd-44a1-8d94-cb7d77e399bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.505886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5697b7a9-47cd-44a1-8d94-cb7d77e399bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:21.507870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5697b7a9-47cd-44a1-8d94-cb7d77e399bf admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.522652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 956/1921] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:21 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:21.526649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424ab7c-389f-44c5-91ef-fff117166f1b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.539976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424ab7c-389f-44c5-91ef-fff117166f1b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:21.545338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691512e1-70b7-4b94-9017-2c376bf26d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:21.546369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691512e1-70b7-4b94-9017-2c376bf26d0f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.546647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691512e1-70b7-4b94-9017-2c376bf26d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.546919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691512e1-70b7-4b94-9017-2c376bf26d0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.547566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691512e1-70b7-4b94-9017-2c376bf26d0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:21.549164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691512e1-70b7-4b94-9017-2c376bf26d0f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.562390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 966/1922] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:21 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:21.566996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5697b7a9-47cd-44a1-8d94-cb7d77e399bf admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.583490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5697b7a9-47cd-44a1-8d94-cb7d77e399bf admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:21.588049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd7d534-d652-4ff4-890e-7e6d229ab27f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:21.588699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd7d534-d652-4ff4-890e-7e6d229ab27f admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.588978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd7d534-d652-4ff4-890e-7e6d229ab27f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.589251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd7d534-d652-4ff4-890e-7e6d229ab27f admin admin] PATH_INFO: `/v3/projects/6db6da59f5894fdeaf68de61254b28f6/users/7444475ae92442979967ec34d69a8c28/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.603800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd7d534-d652-4ff4-890e-7e6d229ab27f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6db6da59f5894fdeaf68de61254b28f6, user_id=7444475ae92442979967ec34d69a8c28, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:21.606515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd7d534-d652-4ff4-890e-7e6d229ab27f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.620146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 957/1923] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:21 2022] PUT /identity/v3/projects/6db6da59f5894fdeaf68de61254b28f6/users/7444475ae92442979967ec34d69a8c28/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:21.631946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1217b2a6-4c72-43f3-a194-042558c346d1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.632296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1217b2a6-4c72-43f3-a194-042558c346d1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.632571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1217b2a6-4c72-43f3-a194-042558c346d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.660160 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1217b2a6-4c72-43f3-a194-042558c346d1 None None] MFA Rules not processed for user `7444475ae92442979967ec34d69a8c28`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:21.661345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1217b2a6-4c72-43f3-a194-042558c346d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:21.747044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 967/1924] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:21.751462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd7d534-d652-4ff4-890e-7e6d229ab27f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.811423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd7d534-d652-4ff4-890e-7e6d229ab27f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:21.815915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfff6fc2-fd46-4c62-8347-6921fde4e9a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:21.817119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfff6fc2-fd46-4c62-8347-6921fde4e9a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.817394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfff6fc2-fd46-4c62-8347-6921fde4e9a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.817665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfff6fc2-fd46-4c62-8347-6921fde4e9a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.818318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfff6fc2-fd46-4c62-8347-6921fde4e9a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:21.820196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfff6fc2-fd46-4c62-8347-6921fde4e9a8 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.845318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 958/1925] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:21 2022] POST /identity/v3/projects => generated 377 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:21.849360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1217b2a6-4c72-43f3-a194-042558c346d1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.860188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1217b2a6-4c72-43f3-a194-042558c346d1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:21.940972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1c96b4-a658-4756-bc07-108dd3a75bcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:21.942318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1c96b4-a658-4756-bc07-108dd3a75bcf admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:21.942619 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1c96b4-a658-4756-bc07-108dd3a75bcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:21.942930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d1c96b4-a658-4756-bc07-108dd3a75bcf admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:21.943596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1c96b4-a658-4756-bc07-108dd3a75bcf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:21.945741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d1c96b4-a658-4756-bc07-108dd3a75bcf admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:21.977885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 968/1926] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:21 2022] POST /identity/v3/users => generated 295 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:21.979250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfff6fc2-fd46-4c62-8347-6921fde4e9a8 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:21.997531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfff6fc2-fd46-4c62-8347-6921fde4e9a8 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:22.005594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db0651-c27d-47c7-9e43-2e817233722a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:22.006386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db0651-c27d-47c7-9e43-2e817233722a admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:22.006678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db0651-c27d-47c7-9e43-2e817233722a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:22.006952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db0651-c27d-47c7-9e43-2e817233722a admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:22.007664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db0651-c27d-47c7-9e43-2e817233722a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:22.010015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db0651-c27d-47c7-9e43-2e817233722a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:22.029321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 959/1927] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:21 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:22.034527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1c96b4-a658-4756-bc07-108dd3a75bcf admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:22.047131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d1c96b4-a658-4756-bc07-108dd3a75bcf admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:22.053293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7723a44c-657d-4d11-b6c4-76e8da30e1f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:22.053971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7723a44c-657d-4d11-b6c4-76e8da30e1f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:22.054302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7723a44c-657d-4d11-b6c4-76e8da30e1f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:22.054579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7723a44c-657d-4d11-b6c4-76e8da30e1f9 admin admin] PATH_INFO: `/v3/projects/8d59bc1a29c049368c9126b5a63cd685/users/8072e00f0d764d04a92e635e43f694a2/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:22.080340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:22.084614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7723a44c-657d-4d11-b6c4-76e8da30e1f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d59bc1a29c049368c9126b5a63cd685, user_id=8072e00f0d764d04a92e635e43f694a2, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:22.088382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7723a44c-657d-4d11-b6c4-76e8da30e1f9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:22.106858 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 969/1928] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:22 2022] PUT /identity/v3/projects/8d59bc1a29c049368c9126b5a63cd685/users/8072e00f0d764d04a92e635e43f694a2/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:22.111123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db0651-c27d-47c7-9e43-2e817233722a admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:22.124753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db0651-c27d-47c7-9e43-2e817233722a admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:22.259850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40903a6-c4c9-4697-b979-bcc348cedd5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:22.264968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40903a6-c4c9-4697-b979-bcc348cedd5b admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:22.265446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40903a6-c4c9-4697-b979-bcc348cedd5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:22.265695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40903a6-c4c9-4697-b979-bcc348cedd5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:22.271869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40903a6-c4c9-4697-b979-bcc348cedd5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:22.273498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40903a6-c4c9-4697-b979-bcc348cedd5b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:22.292717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 960/1929] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:22 2022] GET /identity/v3/roles => generated 1420 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:22.293460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7723a44c-657d-4d11-b6c4-76e8da30e1f9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:22.324060 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7723a44c-657d-4d11-b6c4-76e8da30e1f9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:22.333482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdd9e29-03b0-4e39-b6c9-603abb88027e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:22.334201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdd9e29-03b0-4e39-b6c9-603abb88027e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:22.334483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdd9e29-03b0-4e39-b6c9-603abb88027e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:22.334760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdd9e29-03b0-4e39-b6c9-603abb88027e admin admin] PATH_INFO: `/v3/projects/8d59bc1a29c049368c9126b5a63cd685/users/8072e00f0d764d04a92e635e43f694a2/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:22.337413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdd9e29-03b0-4e39-b6c9-603abb88027e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d59bc1a29c049368c9126b5a63cd685, user_id=8072e00f0d764d04a92e635e43f694a2, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:22.340895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdd9e29-03b0-4e39-b6c9-603abb88027e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:22.376043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 970/1930] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:22 2022] PUT /identity/v3/projects/8d59bc1a29c049368c9126b5a63cd685/users/8072e00f0d764d04a92e635e43f694a2/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:22.386279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb421bea-8c50-4bdf-be1c-47023f17b175 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:22.386615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb421bea-8c50-4bdf-be1c-47023f17b175 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:22.387420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb421bea-8c50-4bdf-be1c-47023f17b175 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:22.398717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:22.431323 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-fb421bea-8c50-4bdf-be1c-47023f17b175 None None] MFA Rules not processed for user `8072e00f0d764d04a92e635e43f694a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:22.433619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fb421bea-8c50-4bdf-be1c-47023f17b175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:22.544753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 961/1931] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:22 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:22.556389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdd9e29-03b0-4e39-b6c9-603abb88027e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:22.558495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8fdd9e29-03b0-4e39-b6c9-603abb88027e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:22.639931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdd9e29-03b0-4e39-b6c9-603abb88027e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:22.645705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3eab320-a43b-4594-8e72-c3d4adba7871 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:22.646569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eab320-a43b-4594-8e72-c3d4adba7871 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:22.646967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eab320-a43b-4594-8e72-c3d4adba7871 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:22.647355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3eab320-a43b-4594-8e72-c3d4adba7871 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:22.720477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3eab320-a43b-4594-8e72-c3d4adba7871 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:22.721935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3eab320-a43b-4594-8e72-c3d4adba7871 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:22.733483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 971/1932] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:22 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:23.796758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb421bea-8c50-4bdf-be1c-47023f17b175 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:23.798680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fb421bea-8c50-4bdf-be1c-47023f17b175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:23.882988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb421bea-8c50-4bdf-be1c-47023f17b175 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:23.888995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:23.889936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:23.890477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:23.890937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] PATH_INFO: `/v3/users/c213b79123bb4fd19018aa1056bc4c66` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:23.892504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c213b79123bb4fd19018aa1056bc4c66)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:23.895880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:23.972067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c213b79123bb4fd19018aa1056bc4c66'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:23.973552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c213b79123bb4fd19018aa1056bc4c66'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:23.990913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c213b79123bb4fd19018aa1056bc4c66'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:23.992448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c213b79123bb4fd19018aa1056bc4c66'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:24.006284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c213b79123bb4fd19018aa1056bc4c66'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:24.068801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 962/1933] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:23 2022] DELETE /identity/v3/users/c213b79123bb4fd19018aa1056bc4c66 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:24.077018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a1ad84-516d-4fb0-96f1-8cce413ed97a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:24.077468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a1ad84-516d-4fb0-96f1-8cce413ed97a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:24.077614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a1ad84-516d-4fb0-96f1-8cce413ed97a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:24.106981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-65a1ad84-516d-4fb0-96f1-8cce413ed97a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:24.108783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-65a1ad84-516d-4fb0-96f1-8cce413ed97a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:24.199273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 972/1934] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:24.215402 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:24.217307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:24.230410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:24.349687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f96d21-609c-45ed-946d-03c983674e30 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:24.358044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4c57c2-0df7-489f-8d04-309abba53a2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:24.359084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4c57c2-0df7-489f-8d04-309abba53a2c service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:24.361837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4c57c2-0df7-489f-8d04-309abba53a2c service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:24.362251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4c57c2-0df7-489f-8d04-309abba53a2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:24.384042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a1ad84-516d-4fb0-96f1-8cce413ed97a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:24.385769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-65a1ad84-516d-4fb0-96f1-8cce413ed97a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:24.388664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4c57c2-0df7-489f-8d04-309abba53a2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:24.390464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4c57c2-0df7-489f-8d04-309abba53a2c service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:24.401658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 963/1935] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:24.438708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:24.472907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a1ad84-516d-4fb0-96f1-8cce413ed97a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:24.483948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614d9598-42e8-4d36-9eab-f2ca18aeb50c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:24.484823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614d9598-42e8-4d36-9eab-f2ca18aeb50c service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:24.485190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614d9598-42e8-4d36-9eab-f2ca18aeb50c service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:24.485532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614d9598-42e8-4d36-9eab-f2ca18aeb50c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:24.487409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-614d9598-42e8-4d36-9eab-f2ca18aeb50c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:24.562935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614d9598-42e8-4d36-9eab-f2ca18aeb50c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:24.564609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614d9598-42e8-4d36-9eab-f2ca18aeb50c service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:24.575162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 973/1936] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:24 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:24.783960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4c57c2-0df7-489f-8d04-309abba53a2c service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:24.786205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4e4c57c2-0df7-489f-8d04-309abba53a2c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:24.804021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4c57c2-0df7-489f-8d04-309abba53a2c service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:24.820486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:24.823155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:24.823516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:24.823866 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] PATH_INFO: `/v3/projects/f312db0eb0094badbcdd306789b776d2` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:24.825204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] RBAC: Authorizing `identity:delete_project(project_id=f312db0eb0094badbcdd306789b776d2)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:24.827248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:24.898564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f312db0eb0094badbcdd306789b776d2', 'request_id': 'req-d4ad3fd8-b783-41c3-923b-518f572a2fba'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:24.899629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614d9598-42e8-4d36-9eab-f2ca18aeb50c service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:24.901119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-614d9598-42e8-4d36-9eab-f2ca18aeb50c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:24.908600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f312db0eb0094badbcdd306789b776d2', 'request_id': 'req-d4ad3fd8-b783-41c3-923b-518f572a2fba'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:24.914330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614d9598-42e8-4d36-9eab-f2ca18aeb50c service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:24.917916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f312db0eb0094badbcdd306789b776d2', 'request_id': 'req-d4ad3fd8-b783-41c3-923b-518f572a2fba'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:24.931129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] The token cache is being invalidate because project f312db0eb0094badbcdd306789b776d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:24.931505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:24.932154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:24.933656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 964/1937] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:24 2022] DELETE /identity/v3/projects/f312db0eb0094badbcdd306789b776d2 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:24.937864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:24.938961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:24.967360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e023600-19ec-4b67-83a7-3aa5cb6b2fb1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:24.967951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e023600-19ec-4b67-83a7-3aa5cb6b2fb1 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:24.968251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e023600-19ec-4b67-83a7-3aa5cb6b2fb1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:24.968517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e023600-19ec-4b67-83a7-3aa5cb6b2fb1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:24.970476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5e023600-19ec-4b67-83a7-3aa5cb6b2fb1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:25.002433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ad3fd8-b783-41c3-923b-518f572a2fba admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:25.007103 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:25.007730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:25.008010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:25.008286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] PATH_INFO: `/v3/users/33894bc40e3f4deeb549e7293b456b1b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:25.009397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=33894bc40e3f4deeb549e7293b456b1b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:25.011351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:25.030901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e023600-19ec-4b67-83a7-3aa5cb6b2fb1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:25.032373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e023600-19ec-4b67-83a7-3aa5cb6b2fb1 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:25.043029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 974/1938] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:24 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:25.052735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:25.062990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33894bc40e3f4deeb549e7293b456b1b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:25.064072 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33894bc40e3f4deeb549e7293b456b1b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:25.071275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33894bc40e3f4deeb549e7293b456b1b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:25.071947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33894bc40e3f4deeb549e7293b456b1b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:25.079374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-67398142-9afc-4c81-942c-c14d476fdee4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33894bc40e3f4deeb549e7293b456b1b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:25.098269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 965/1939] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:24 2022] DELETE /identity/v3/users/33894bc40e3f4deeb549e7293b456b1b => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:25.379307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e023600-19ec-4b67-83a7-3aa5cb6b2fb1 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:25.394851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e023600-19ec-4b67-83a7-3aa5cb6b2fb1 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:25.441288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:25.460287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:25.460996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:25.461282 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:25.461554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] PATH_INFO: `/v3/projects/f7ff3986cae84b2f92a36f86c384411e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:25.462726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7ff3986cae84b2f92a36f86c384411e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:25.464708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:25.515438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7ff3986cae84b2f92a36f86c384411e', 'request_id': 'req-e154badb-89f5-4dca-a4d9-338c1e691da7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:25.524064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7ff3986cae84b2f92a36f86c384411e', 'request_id': 'req-e154badb-89f5-4dca-a4d9-338c1e691da7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:25.531092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7ff3986cae84b2f92a36f86c384411e', 'request_id': 'req-e154badb-89f5-4dca-a4d9-338c1e691da7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:25.546389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] The token cache is being invalidate because project f7ff3986cae84b2f92a36f86c384411e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:25.546683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:25.547353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e154badb-89f5-4dca-a4d9-338c1e691da7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:25.552114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 975/1940] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:25 2022] DELETE /identity/v3/projects/f7ff3986cae84b2f92a36f86c384411e => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:25.553565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcda1af2-d815-4a23-803a-b1a4581b02cd None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:25.553898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcda1af2-d815-4a23-803a-b1a4581b02cd None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:25.554157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcda1af2-d815-4a23-803a-b1a4581b02cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:25.577514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-fcda1af2-d815-4a23-803a-b1a4581b02cd None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:25.578807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fcda1af2-d815-4a23-803a-b1a4581b02cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:25.641828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 966/1941] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:25.650086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f4d2f9-edb4-4548-a947-428901523fe2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:25.650444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f4d2f9-edb4-4548-a947-428901523fe2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:25.650787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f4d2f9-edb4-4548-a947-428901523fe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:25.671592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c1f4d2f9-edb4-4548-a947-428901523fe2 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:25.672805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c1f4d2f9-edb4-4548-a947-428901523fe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:25.680852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 976/1942] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:25.684896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcda1af2-d815-4a23-803a-b1a4581b02cd None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:25.694708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcda1af2-d815-4a23-803a-b1a4581b02cd None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:25.699636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28534d56-a697-4256-b018-7e2c5c8dff55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:25.700245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28534d56-a697-4256-b018-7e2c5c8dff55 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:25.700520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28534d56-a697-4256-b018-7e2c5c8dff55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:25.700792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28534d56-a697-4256-b018-7e2c5c8dff55 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:25.701471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28534d56-a697-4256-b018-7e2c5c8dff55 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:25.701798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28534d56-a697-4256-b018-7e2c5c8dff55 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:25.703417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28534d56-a697-4256-b018-7e2c5c8dff55 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:25.721842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 967/1943] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:25.726453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f4d2f9-edb4-4548-a947-428901523fe2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:25.735792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f4d2f9-edb4-4548-a947-428901523fe2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:25.740865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a8987a-70dd-4c95-bced-9e6c7e93dffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:25.741467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a8987a-70dd-4c95-bced-9e6c7e93dffc admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:25.741745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a8987a-70dd-4c95-bced-9e6c7e93dffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:25.742036 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a8987a-70dd-4c95-bced-9e6c7e93dffc admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:25.742707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a8987a-70dd-4c95-bced-9e6c7e93dffc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:25.744667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a8987a-70dd-4c95-bced-9e6c7e93dffc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:25.774385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 977/1944] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:25 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:25.776751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28534d56-a697-4256-b018-7e2c5c8dff55 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:25.787489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28534d56-a697-4256-b018-7e2c5c8dff55 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:25.848963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d99a3d-8306-46f4-ba01-18d6369a6aec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:25.849595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d99a3d-8306-46f4-ba01-18d6369a6aec admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:25.849762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d99a3d-8306-46f4-ba01-18d6369a6aec admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:25.850036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d99a3d-8306-46f4-ba01-18d6369a6aec admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:25.850713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d99a3d-8306-46f4-ba01-18d6369a6aec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:25.852640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d99a3d-8306-46f4-ba01-18d6369a6aec admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:25.874080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 968/1945] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:25 2022] POST /identity/v3/users => generated 292 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:25.878358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a8987a-70dd-4c95-bced-9e6c7e93dffc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:25.889116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a8987a-70dd-4c95-bced-9e6c7e93dffc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:25.893710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcee7ce7-cb22-4979-8398-6f649d0484e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:25.894955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcee7ce7-cb22-4979-8398-6f649d0484e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:25.895241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcee7ce7-cb22-4979-8398-6f649d0484e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:25.895512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcee7ce7-cb22-4979-8398-6f649d0484e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:25.896181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcee7ce7-cb22-4979-8398-6f649d0484e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:25.897761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcee7ce7-cb22-4979-8398-6f649d0484e7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:25.910135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 978/1946] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:25.911674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d99a3d-8306-46f4-ba01-18d6369a6aec admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:25.921552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d99a3d-8306-46f4-ba01-18d6369a6aec admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:25.926240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac9bc96-08a3-42b1-8034-13b72a2d693a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:25.926832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac9bc96-08a3-42b1-8034-13b72a2d693a admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:25.927113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac9bc96-08a3-42b1-8034-13b72a2d693a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:25.927384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac9bc96-08a3-42b1-8034-13b72a2d693a admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:25.928016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac9bc96-08a3-42b1-8034-13b72a2d693a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:25.929584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac9bc96-08a3-42b1-8034-13b72a2d693a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:25.940993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 969/1947] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:25.945637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcee7ce7-cb22-4979-8398-6f649d0484e7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:25.955311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcee7ce7-cb22-4979-8398-6f649d0484e7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:25.960515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a540bc6f-df21-4d0b-8b54-3ff7f1959431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:25.961155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a540bc6f-df21-4d0b-8b54-3ff7f1959431 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:25.961429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a540bc6f-df21-4d0b-8b54-3ff7f1959431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:25.961699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a540bc6f-df21-4d0b-8b54-3ff7f1959431 admin admin] PATH_INFO: `/v3/projects/9060b3b6ec4b4af883bc0f491fa4db29/users/a629feef960346398354708690a83ae6/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:25.975722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a540bc6f-df21-4d0b-8b54-3ff7f1959431 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9060b3b6ec4b4af883bc0f491fa4db29, user_id=a629feef960346398354708690a83ae6, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:25.978383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a540bc6f-df21-4d0b-8b54-3ff7f1959431 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:25.990527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 979/1948] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:25 2022] PUT /identity/v3/projects/9060b3b6ec4b4af883bc0f491fa4db29/users/a629feef960346398354708690a83ae6/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:26.001110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f6cfa1-57ac-41bd-8bf5-403efa2e644d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:26.001391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f6cfa1-57ac-41bd-8bf5-403efa2e644d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:26.001662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f6cfa1-57ac-41bd-8bf5-403efa2e644d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:26.028911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e0f6cfa1-57ac-41bd-8bf5-403efa2e644d None None] MFA Rules not processed for user `a629feef960346398354708690a83ae6`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:26.030150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e0f6cfa1-57ac-41bd-8bf5-403efa2e644d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:26.039803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:26.096899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 970/1949] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:25 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:26.101476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a540bc6f-df21-4d0b-8b54-3ff7f1959431 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:26.161913 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a540bc6f-df21-4d0b-8b54-3ff7f1959431 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:26.166570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1283b2f7-00f1-4ed7-bfb9-f5e749123e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:26.167167 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1283b2f7-00f1-4ed7-bfb9-f5e749123e5e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:26.167445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1283b2f7-00f1-4ed7-bfb9-f5e749123e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:26.167730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1283b2f7-00f1-4ed7-bfb9-f5e749123e5e admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:26.168350 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1283b2f7-00f1-4ed7-bfb9-f5e749123e5e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:26.170283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1283b2f7-00f1-4ed7-bfb9-f5e749123e5e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:26.184199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:26.197589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 980/1950] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:26 2022] POST /identity/v3/projects => generated 383 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:26.201617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f6cfa1-57ac-41bd-8bf5-403efa2e644d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:26.212643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f6cfa1-57ac-41bd-8bf5-403efa2e644d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:26.266812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dc0140-76f0-4167-97f2-b712d628c6b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:26.267416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc0140-76f0-4167-97f2-b712d628c6b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:26.267699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc0140-76f0-4167-97f2-b712d628c6b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:26.267974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc0140-76f0-4167-97f2-b712d628c6b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:26.268637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dc0140-76f0-4167-97f2-b712d628c6b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:26.271172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dc0140-76f0-4167-97f2-b712d628c6b3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:26.291221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 971/1951] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:26 2022] POST /identity/v3/users => generated 298 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:26.295141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1283b2f7-00f1-4ed7-bfb9-f5e749123e5e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:26.305811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1283b2f7-00f1-4ed7-bfb9-f5e749123e5e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:26.310682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f3774d-09ea-42ac-ba3c-3be38424a466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:26.311279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f3774d-09ea-42ac-ba3c-3be38424a466 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:26.311557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f3774d-09ea-42ac-ba3c-3be38424a466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:26.311828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f3774d-09ea-42ac-ba3c-3be38424a466 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:26.312460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f3774d-09ea-42ac-ba3c-3be38424a466 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:26.314189 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f3774d-09ea-42ac-ba3c-3be38424a466 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:26.325156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 981/1952] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:26.329074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dc0140-76f0-4167-97f2-b712d628c6b3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:26.338683 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dc0140-76f0-4167-97f2-b712d628c6b3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:26.343176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed60a630-a7f6-4726-b474-48e261e9727d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:26.343819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed60a630-a7f6-4726-b474-48e261e9727d admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:26.344098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed60a630-a7f6-4726-b474-48e261e9727d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:26.344374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed60a630-a7f6-4726-b474-48e261e9727d admin admin] PATH_INFO: `/v3/projects/643e63a9e0154da1bc4f0f2d1c5c7f56/users/2e35f8dab49e483c963b1c5c124f05da/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:26.358715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed60a630-a7f6-4726-b474-48e261e9727d admin admin] RBAC: Authorizing `identity:create_grant(project_id=643e63a9e0154da1bc4f0f2d1c5c7f56, user_id=2e35f8dab49e483c963b1c5c124f05da, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:26.361306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed60a630-a7f6-4726-b474-48e261e9727d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:26.374596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 972/1953] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:26 2022] PUT /identity/v3/projects/643e63a9e0154da1bc4f0f2d1c5c7f56/users/2e35f8dab49e483c963b1c5c124f05da/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:26.378436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f3774d-09ea-42ac-ba3c-3be38424a466 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:26.388796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f3774d-09ea-42ac-ba3c-3be38424a466 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:26.446909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1f2975-dceb-4ee6-86ac-d004015c470d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:26.447516 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1f2975-dceb-4ee6-86ac-d004015c470d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:26.447794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1f2975-dceb-4ee6-86ac-d004015c470d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:26.448065 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1f2975-dceb-4ee6-86ac-d004015c470d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:26.448692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1f2975-dceb-4ee6-86ac-d004015c470d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:26.450315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1f2975-dceb-4ee6-86ac-d004015c470d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:26.459806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 982/1954] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:26 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:26.464022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed60a630-a7f6-4726-b474-48e261e9727d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:26.474045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed60a630-a7f6-4726-b474-48e261e9727d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:26.478966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b652f993-8ef4-4e96-8f6d-3c268de422f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:26.479649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652f993-8ef4-4e96-8f6d-3c268de422f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:26.479950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652f993-8ef4-4e96-8f6d-3c268de422f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:26.480227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652f993-8ef4-4e96-8f6d-3c268de422f3 admin admin] PATH_INFO: `/v3/projects/643e63a9e0154da1bc4f0f2d1c5c7f56/users/2e35f8dab49e483c963b1c5c124f05da/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:26.482190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b652f993-8ef4-4e96-8f6d-3c268de422f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=643e63a9e0154da1bc4f0f2d1c5c7f56, user_id=2e35f8dab49e483c963b1c5c124f05da, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:26.484875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b652f993-8ef4-4e96-8f6d-3c268de422f3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:26.495394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 973/1955] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:26 2022] PUT /identity/v3/projects/643e63a9e0154da1bc4f0f2d1c5c7f56/users/2e35f8dab49e483c963b1c5c124f05da/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:26.507517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc1c70e-c063-417d-ba64-db61beac094f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:26.507916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc1c70e-c063-417d-ba64-db61beac094f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:26.508199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc1c70e-c063-417d-ba64-db61beac094f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:26.536060 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2cc1c70e-c063-417d-ba64-db61beac094f None None] MFA Rules not processed for user `2e35f8dab49e483c963b1c5c124f05da`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:26.537259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2cc1c70e-c063-417d-ba64-db61beac094f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:26.604230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 983/1956] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:26 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:26.843462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b652f993-8ef4-4e96-8f6d-3c268de422f3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:26.845152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b652f993-8ef4-4e96-8f6d-3c268de422f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:26.903508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b652f993-8ef4-4e96-8f6d-3c268de422f3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:26.907136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fcf80f-54e3-4e16-9e30-9c288b7ae36b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:26.907748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fcf80f-54e3-4e16-9e30-9c288b7ae36b service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:26.908029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fcf80f-54e3-4e16-9e30-9c288b7ae36b service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:26.908303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02fcf80f-54e3-4e16-9e30-9c288b7ae36b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:26.917929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fcf80f-54e3-4e16-9e30-9c288b7ae36b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:26.919391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02fcf80f-54e3-4e16-9e30-9c288b7ae36b service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:26.929530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 974/1957] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:26 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:27.403362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc1c70e-c063-417d-ba64-db61beac094f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:27.412613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc1c70e-c063-417d-ba64-db61beac094f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:27.416078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942d0a9b-cadf-4e48-a045-74fd03e251b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:27.416662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942d0a9b-cadf-4e48-a045-74fd03e251b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:27.416936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942d0a9b-cadf-4e48-a045-74fd03e251b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:27.417201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942d0a9b-cadf-4e48-a045-74fd03e251b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:27.418886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-942d0a9b-cadf-4e48-a045-74fd03e251b6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:27.482773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942d0a9b-cadf-4e48-a045-74fd03e251b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:27.484201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942d0a9b-cadf-4e48-a045-74fd03e251b6 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:27.488051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:27.495116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 984/1958] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:27 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:27.552372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fcf80f-54e3-4e16-9e30-9c288b7ae36b service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:27.557453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:27.560823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02fcf80f-54e3-4e16-9e30-9c288b7ae36b service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:27.565355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925c279c-c202-468b-96c3-b3ffc5d02dcf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:27.566667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925c279c-c202-468b-96c3-b3ffc5d02dcf service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:27.567159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925c279c-c202-468b-96c3-b3ffc5d02dcf service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:27.567486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925c279c-c202-468b-96c3-b3ffc5d02dcf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:27.622152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925c279c-c202-468b-96c3-b3ffc5d02dcf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:27.623564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925c279c-c202-468b-96c3-b3ffc5d02dcf service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:27.632776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 975/1959] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:27 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:30.005084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942d0a9b-cadf-4e48-a045-74fd03e251b6 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:30.006775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-942d0a9b-cadf-4e48-a045-74fd03e251b6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:30.074762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942d0a9b-cadf-4e48-a045-74fd03e251b6 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:30.079978 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:30.080744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:30.081035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:30.082232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] PATH_INFO: `/v3/users/e2209ce29c0e4edd8969c1521061ad2b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:30.083489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e2209ce29c0e4edd8969c1521061ad2b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:30.085435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:30.160612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e2209ce29c0e4edd8969c1521061ad2b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:30.180969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e2209ce29c0e4edd8969c1521061ad2b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:30.181716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e2209ce29c0e4edd8969c1521061ad2b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:30.182364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e2209ce29c0e4edd8969c1521061ad2b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:30.190172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e2209ce29c0e4edd8969c1521061ad2b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:30.209228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 985/1960] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:30 2022] DELETE /identity/v3/users/e2209ce29c0e4edd8969c1521061ad2b => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:30.213762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f6195-db82-4300-b8bf-6e72210e55e5 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:30.214296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f6195-db82-4300-b8bf-6e72210e55e5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:30.214403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7f6195-db82-4300-b8bf-6e72210e55e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:30.233060 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-2f7f6195-db82-4300-b8bf-6e72210e55e5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:30.234289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2f7f6195-db82-4300-b8bf-6e72210e55e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:30.293863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 976/1961] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:30.304064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:30.305447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:30.360705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc7982c-56f0-401b-b0e1-4709bcb1ffb6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:30.364819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c092e2-f4ed-408c-9051-7fe8df5e58e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:30.365357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c092e2-f4ed-408c-9051-7fe8df5e58e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:30.365632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c092e2-f4ed-408c-9051-7fe8df5e58e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:30.365901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c092e2-f4ed-408c-9051-7fe8df5e58e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:30.375000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c092e2-f4ed-408c-9051-7fe8df5e58e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:30.376413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c092e2-f4ed-408c-9051-7fe8df5e58e1 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:30.386306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 986/1962] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:30.815090 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f6195-db82-4300-b8bf-6e72210e55e5 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:30.816640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2f7f6195-db82-4300-b8bf-6e72210e55e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:30.828946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7f6195-db82-4300-b8bf-6e72210e55e5 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:30.836060 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:30.836777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:30.838140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:30.838140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] PATH_INFO: `/v3/projects/84d26c9884eb46b9967b2f34bf1f212a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:30.838678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=84d26c9884eb46b9967b2f34bf1f212a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:30.840771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:30.899089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84d26c9884eb46b9967b2f34bf1f212a', 'request_id': 'req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:30.906155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84d26c9884eb46b9967b2f34bf1f212a', 'request_id': 'req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:30.912946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84d26c9884eb46b9967b2f34bf1f212a', 'request_id': 'req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:30.923190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] The token cache is being invalidate because project 84d26c9884eb46b9967b2f34bf1f212a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:30.923474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:30.924096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:30.925596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 977/1963] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:30 2022] DELETE /identity/v3/projects/84d26c9884eb46b9967b2f34bf1f212a => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:30.929638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c092e2-f4ed-408c-9051-7fe8df5e58e1 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:30.939013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c092e2-f4ed-408c-9051-7fe8df5e58e1 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:30.954863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:31.002459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:31.003089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:31.003369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:31.003641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] PATH_INFO: `/v3/users/7f8238108f4e4b8f98b2c4d6c901184c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:31.004747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f8238108f4e4b8f98b2c4d6c901184c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:31.006715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:31.054053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f8238108f4e4b8f98b2c4d6c901184c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.060819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f8238108f4e4b8f98b2c4d6c901184c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.061502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f8238108f4e4b8f98b2c4d6c901184c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.062145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f8238108f4e4b8f98b2c4d6c901184c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.069739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f8238108f4e4b8f98b2c4d6c901184c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.086445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 987/1964] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:30 2022] DELETE /identity/v3/users/7f8238108f4e4b8f98b2c4d6c901184c => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:31.349555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:31.351071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:31.366032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95ee7c5-324a-46ca-86cf-26bb35e03ae8 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:31.391115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:31.423003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:31.423622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:31.423899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:31.424166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] PATH_INFO: `/v3/projects/b7c1d2e23f1c47fea1b528fb9365d583` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:31.425296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7c1d2e23f1c47fea1b528fb9365d583)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:31.427907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:31.475541 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7c1d2e23f1c47fea1b528fb9365d583', 'request_id': 'req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.482198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7c1d2e23f1c47fea1b528fb9365d583', 'request_id': 'req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.488943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7c1d2e23f1c47fea1b528fb9365d583', 'request_id': 'req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.499142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] The token cache is being invalidate because project b7c1d2e23f1c47fea1b528fb9365d583 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:31.499425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.500054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.501515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 978/1965] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:31 2022] DELETE /identity/v3/projects/b7c1d2e23f1c47fea1b528fb9365d583 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:31.505490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:31.506913 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:31.519750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8e8a24-84c8-4958-88a3-343a0138d943 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:31.574621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:31.575344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:31.575627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:31.575904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] PATH_INFO: `/v3/users/f9d9424e9f014ef69c70bb7275ffefb1` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:31.577010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] RBAC: Authorizing `identity:delete_user(user_id=f9d9424e9f014ef69c70bb7275ffefb1)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:31.578957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:31.619309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:31.625251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f9d9424e9f014ef69c70bb7275ffefb1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.633716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f9d9424e9f014ef69c70bb7275ffefb1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.634404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f9d9424e9f014ef69c70bb7275ffefb1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.635121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f9d9424e9f014ef69c70bb7275ffefb1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.642417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-602718bc-12e3-42a4-9019-7bc4fc4c4abd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f9d9424e9f014ef69c70bb7275ffefb1'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:31.660574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 988/1966] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:31 2022] DELETE /identity/v3/users/f9d9424e9f014ef69c70bb7275ffefb1 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:32.055420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:32.056849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:32.070005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d403e149-1ae6-40f2-9bc0-cea4bc8de96d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:32.158685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:32.159319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.159624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.159898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] PATH_INFO: `/v3/projects/4abd4fec37f64eaea237212dbf23783a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.161008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4abd4fec37f64eaea237212dbf23783a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:32.162924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:32.208446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4abd4fec37f64eaea237212dbf23783a', 'request_id': 'req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:32.215766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4abd4fec37f64eaea237212dbf23783a', 'request_id': 'req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:32.217627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:32.223327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4abd4fec37f64eaea237212dbf23783a', 'request_id': 'req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:32.234408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] The token cache is being invalidate because project 4abd4fec37f64eaea237212dbf23783a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:32.234718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:32.235350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0bb47ce4-e1af-4c0b-a488-e8e3c939be42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:32.236818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 979/1967] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:32 2022] DELETE /identity/v3/projects/4abd4fec37f64eaea237212dbf23783a => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:32.242593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc190db-721b-4ba0-a700-e2b94006e855 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.243041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc190db-721b-4ba0-a700-e2b94006e855 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.243126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc190db-721b-4ba0-a700-e2b94006e855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.263894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-8bc190db-721b-4ba0-a700-e2b94006e855 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:32.265126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8bc190db-721b-4ba0-a700-e2b94006e855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:32.325546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 989/1968] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:32.333760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f7a393-9d42-40d0-8e1a-57b55a210eca None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.334043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f7a393-9d42-40d0-8e1a-57b55a210eca None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.334350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f7a393-9d42-40d0-8e1a-57b55a210eca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.356405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-38f7a393-9d42-40d0-8e1a-57b55a210eca None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:32.357708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-38f7a393-9d42-40d0-8e1a-57b55a210eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:32.367304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 980/1969] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:32.372149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc190db-721b-4ba0-a700-e2b94006e855 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:32.382140 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc190db-721b-4ba0-a700-e2b94006e855 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:32.386654 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80796a80-b921-4305-9d3f-e67ab9784b90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:32.387248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80796a80-b921-4305-9d3f-e67ab9784b90 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.387525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80796a80-b921-4305-9d3f-e67ab9784b90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.387795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80796a80-b921-4305-9d3f-e67ab9784b90 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.388435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80796a80-b921-4305-9d3f-e67ab9784b90 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:32.388731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80796a80-b921-4305-9d3f-e67ab9784b90 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:32.390394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80796a80-b921-4305-9d3f-e67ab9784b90 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:32.401925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 990/1970] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:55:32.406016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f7a393-9d42-40d0-8e1a-57b55a210eca None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:32.415083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f7a393-9d42-40d0-8e1a-57b55a210eca None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:32.419670 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75743874-0e7d-42f4-b244-1d0c5433ddff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:32.420268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75743874-0e7d-42f4-b244-1d0c5433ddff admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.420550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75743874-0e7d-42f4-b244-1d0c5433ddff admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.420833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75743874-0e7d-42f4-b244-1d0c5433ddff admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.421461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75743874-0e7d-42f4-b244-1d0c5433ddff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:32.423389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75743874-0e7d-42f4-b244-1d0c5433ddff admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:32.451875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 981/1971] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:32 2022] POST /identity/v3/projects => generated 367 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:32.455993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80796a80-b921-4305-9d3f-e67ab9784b90 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:32.465907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80796a80-b921-4305-9d3f-e67ab9784b90 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:32.523150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9643ba-fcd5-469d-952c-94566dc045b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:32.523753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9643ba-fcd5-469d-952c-94566dc045b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.524033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9643ba-fcd5-469d-952c-94566dc045b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.524304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9643ba-fcd5-469d-952c-94566dc045b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.524939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9643ba-fcd5-469d-952c-94566dc045b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:32.526892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9643ba-fcd5-469d-952c-94566dc045b2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:32.548946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 991/1972] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:32 2022] POST /identity/v3/users => generated 284 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:32.553948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75743874-0e7d-42f4-b244-1d0c5433ddff admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:32.563974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75743874-0e7d-42f4-b244-1d0c5433ddff admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:32.568975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a11527-736d-4a9f-ae4f-daced1f4d117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:32.569567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a11527-736d-4a9f-ae4f-daced1f4d117 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.569890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a11527-736d-4a9f-ae4f-daced1f4d117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.570190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a11527-736d-4a9f-ae4f-daced1f4d117 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.570874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a11527-736d-4a9f-ae4f-daced1f4d117 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:32.572489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a11527-736d-4a9f-ae4f-daced1f4d117 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:32.582510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 982/1973] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:32.586401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9643ba-fcd5-469d-952c-94566dc045b2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:32.596181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9643ba-fcd5-469d-952c-94566dc045b2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:32.600685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba61f52-7a1b-41ae-920d-cd2eb06a3c56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:32.601276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba61f52-7a1b-41ae-920d-cd2eb06a3c56 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.601550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba61f52-7a1b-41ae-920d-cd2eb06a3c56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.601827 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba61f52-7a1b-41ae-920d-cd2eb06a3c56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.602497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba61f52-7a1b-41ae-920d-cd2eb06a3c56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:32.604071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba61f52-7a1b-41ae-920d-cd2eb06a3c56 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:32.614834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 992/1974] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:32.618775 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a11527-736d-4a9f-ae4f-daced1f4d117 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:32.629435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a11527-736d-4a9f-ae4f-daced1f4d117 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:32.634891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5066253a-15fd-4e6a-b44c-ab5f0271ec57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:32.635533 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5066253a-15fd-4e6a-b44c-ab5f0271ec57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.635810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5066253a-15fd-4e6a-b44c-ab5f0271ec57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.636079 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5066253a-15fd-4e6a-b44c-ab5f0271ec57 admin admin] PATH_INFO: `/v3/projects/548745ac31cf449bbbad9dff9e3cd4f1/users/ce8b1e6dec944b068173bb8c920efe5a/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.651501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5066253a-15fd-4e6a-b44c-ab5f0271ec57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=548745ac31cf449bbbad9dff9e3cd4f1, user_id=ce8b1e6dec944b068173bb8c920efe5a, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:32.655084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5066253a-15fd-4e6a-b44c-ab5f0271ec57 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:32.668627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 983/1975] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:32 2022] PUT /identity/v3/projects/548745ac31cf449bbbad9dff9e3cd4f1/users/ce8b1e6dec944b068173bb8c920efe5a/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:32.681367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091388a3-8fb2-42e6-af57-52e146ea21da None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.681649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091388a3-8fb2-42e6-af57-52e146ea21da None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.682001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091388a3-8fb2-42e6-af57-52e146ea21da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.711664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-091388a3-8fb2-42e6-af57-52e146ea21da None None] MFA Rules not processed for user `ce8b1e6dec944b068173bb8c920efe5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:32.712865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-091388a3-8fb2-42e6-af57-52e146ea21da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:32.724696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:32.778281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 993/1976] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:32 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:32.782722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5066253a-15fd-4e6a-b44c-ab5f0271ec57 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:32.842066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5066253a-15fd-4e6a-b44c-ab5f0271ec57 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:32.848046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b9b344-7454-4582-a87f-082c0fb21ab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:32.848644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9b344-7454-4582-a87f-082c0fb21ab0 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.848921 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9b344-7454-4582-a87f-082c0fb21ab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.849188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9b344-7454-4582-a87f-082c0fb21ab0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.849804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b9b344-7454-4582-a87f-082c0fb21ab0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:32.851761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b9b344-7454-4582-a87f-082c0fb21ab0 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:32.877666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 984/1977] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:32 2022] POST /identity/v3/projects => generated 367 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:32.881532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091388a3-8fb2-42e6-af57-52e146ea21da None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:32.892236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091388a3-8fb2-42e6-af57-52e146ea21da None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:32.953661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e465378-c0b7-4601-8c21-2ced62163632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:32.954316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e465378-c0b7-4601-8c21-2ced62163632 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.954596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e465378-c0b7-4601-8c21-2ced62163632 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.954865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e465378-c0b7-4601-8c21-2ced62163632 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:32.955521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e465378-c0b7-4601-8c21-2ced62163632 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:32.957426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e465378-c0b7-4601-8c21-2ced62163632 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:32.982181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 994/1978] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:32 2022] POST /identity/v3/users => generated 290 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:32.983722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b9b344-7454-4582-a87f-082c0fb21ab0 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:32.994127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b9b344-7454-4582-a87f-082c0fb21ab0 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:32.998762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99826505-5950-47c4-9011-8d4a4a22dd72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:32.999380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99826505-5950-47c4-9011-8d4a4a22dd72 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:32.999656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99826505-5950-47c4-9011-8d4a4a22dd72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:32.999925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99826505-5950-47c4-9011-8d4a4a22dd72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:33.000551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99826505-5950-47c4-9011-8d4a4a22dd72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:33.002153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99826505-5950-47c4-9011-8d4a4a22dd72 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:33.012286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 985/1979] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:32 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:33.016150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e465378-c0b7-4601-8c21-2ced62163632 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:33.026514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e465378-c0b7-4601-8c21-2ced62163632 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:33.031177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2737021-7fab-49ac-a817-45444f8842b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:33.031770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2737021-7fab-49ac-a817-45444f8842b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:33.032050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2737021-7fab-49ac-a817-45444f8842b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:33.032317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2737021-7fab-49ac-a817-45444f8842b7 admin admin] PATH_INFO: `/v3/projects/b8258ea4cfb9402dbac50d953fd2cbf8/users/52086fdaf1334c42a5a3503a0bcf64c7/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:33.047501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2737021-7fab-49ac-a817-45444f8842b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8258ea4cfb9402dbac50d953fd2cbf8, user_id=52086fdaf1334c42a5a3503a0bcf64c7, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:33.050733 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2737021-7fab-49ac-a817-45444f8842b7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:33.064191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 995/1980] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:33 2022] PUT /identity/v3/projects/b8258ea4cfb9402dbac50d953fd2cbf8/users/52086fdaf1334c42a5a3503a0bcf64c7/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:33.068152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99826505-5950-47c4-9011-8d4a4a22dd72 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:33.078378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99826505-5950-47c4-9011-8d4a4a22dd72 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:33.134214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:33.142766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd4d16e-45a1-419e-a22c-c00164e3467a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:33.143535 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd4d16e-45a1-419e-a22c-c00164e3467a admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:33.144737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd4d16e-45a1-419e-a22c-c00164e3467a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:33.144936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd4d16e-45a1-419e-a22c-c00164e3467a admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:33.146432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd4d16e-45a1-419e-a22c-c00164e3467a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:33.148856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd4d16e-45a1-419e-a22c-c00164e3467a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:33.160895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 986/1981] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:33 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:33.166410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2737021-7fab-49ac-a817-45444f8842b7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:33.179984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2737021-7fab-49ac-a817-45444f8842b7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:33.187583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e375846-a3cd-47a7-98e3-f4894a837f59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:33.188935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e375846-a3cd-47a7-98e3-f4894a837f59 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:33.189384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e375846-a3cd-47a7-98e3-f4894a837f59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:33.189826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e375846-a3cd-47a7-98e3-f4894a837f59 admin admin] PATH_INFO: `/v3/projects/b8258ea4cfb9402dbac50d953fd2cbf8/users/52086fdaf1334c42a5a3503a0bcf64c7/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:33.193408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e375846-a3cd-47a7-98e3-f4894a837f59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8258ea4cfb9402dbac50d953fd2cbf8, user_id=52086fdaf1334c42a5a3503a0bcf64c7, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:33.196591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e375846-a3cd-47a7-98e3-f4894a837f59 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:33.210586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 996/1982] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:33 2022] PUT /identity/v3/projects/b8258ea4cfb9402dbac50d953fd2cbf8/users/52086fdaf1334c42a5a3503a0bcf64c7/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:33.222131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9a671b-664b-41f9-9b79-5817ec237568 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:33.222432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9a671b-664b-41f9-9b79-5817ec237568 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:33.222712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9a671b-664b-41f9-9b79-5817ec237568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:33.257380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ff9a671b-664b-41f9-9b79-5817ec237568 None None] MFA Rules not processed for user `52086fdaf1334c42a5a3503a0bcf64c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:33.258620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ff9a671b-664b-41f9-9b79-5817ec237568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:33.377418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 987/1983] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:33 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:33.385874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e375846-a3cd-47a7-98e3-f4894a837f59 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:33.387818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1e375846-a3cd-47a7-98e3-f4894a837f59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:33.483227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e375846-a3cd-47a7-98e3-f4894a837f59 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:33.487307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf0a253-aaad-4c79-acef-da6624a75e4b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:33.487489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf0a253-aaad-4c79-acef-da6624a75e4b service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:33.487808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf0a253-aaad-4c79-acef-da6624a75e4b service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:33.488106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf0a253-aaad-4c79-acef-da6624a75e4b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:33.497497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf0a253-aaad-4c79-acef-da6624a75e4b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:33.498943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf0a253-aaad-4c79-acef-da6624a75e4b service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:33.509014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 997/1984] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:33 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:33.593297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9a671b-664b-41f9-9b79-5817ec237568 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:33.603529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9a671b-664b-41f9-9b79-5817ec237568 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:33.607126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b49c01-b224-42db-a064-326e17605d8c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:33.607732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b49c01-b224-42db-a064-326e17605d8c service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:33.608010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b49c01-b224-42db-a064-326e17605d8c service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:33.608287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b49c01-b224-42db-a064-326e17605d8c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:33.668286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b49c01-b224-42db-a064-326e17605d8c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:33.669703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b49c01-b224-42db-a064-326e17605d8c service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:33.678586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 988/1985] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:33 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:37.178709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf0a253-aaad-4c79-acef-da6624a75e4b service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:37.180113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7cf0a253-aaad-4c79-acef-da6624a75e4b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:37.191510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:37.304824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf0a253-aaad-4c79-acef-da6624a75e4b service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:37.309817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:37.310480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:37.310761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:37.311035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] PATH_INFO: `/v3/users/a629feef960346398354708690a83ae6` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:37.312180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] RBAC: Authorizing `identity:delete_user(user_id=a629feef960346398354708690a83ae6)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:37.314247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:37.400283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a629feef960346398354708690a83ae6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:37.411934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a629feef960346398354708690a83ae6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:37.412680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a629feef960346398354708690a83ae6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:37.413302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a629feef960346398354708690a83ae6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:37.424390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a629feef960346398354708690a83ae6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:37.448760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 998/1986] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:37 2022] DELETE /identity/v3/users/a629feef960346398354708690a83ae6 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:37.453109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9828be1a-124e-4c5b-86f4-7c9240a58fb2 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:37.453619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9828be1a-124e-4c5b-86f4-7c9240a58fb2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:37.453887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9828be1a-124e-4c5b-86f4-7c9240a58fb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:37.479702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9828be1a-124e-4c5b-86f4-7c9240a58fb2 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:37.481331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9828be1a-124e-4c5b-86f4-7c9240a58fb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:37.582948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 989/1987] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:37.590987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:37.592550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:37.668012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97621539-80df-4353-a7f7-b983ecf6bbcd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:37.671745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c90ee68-0cec-48e5-8e6f-4a7ce555a553 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:37.672447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c90ee68-0cec-48e5-8e6f-4a7ce555a553 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:37.672724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c90ee68-0cec-48e5-8e6f-4a7ce555a553 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:37.672994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c90ee68-0cec-48e5-8e6f-4a7ce555a553 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:37.684924 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c90ee68-0cec-48e5-8e6f-4a7ce555a553 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:37.686588 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c90ee68-0cec-48e5-8e6f-4a7ce555a553 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:37.697949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 999/1988] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:37.985505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9828be1a-124e-4c5b-86f4-7c9240a58fb2 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:37.988705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9828be1a-124e-4c5b-86f4-7c9240a58fb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:38.000324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9828be1a-124e-4c5b-86f4-7c9240a58fb2 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:38.005484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:38.006338 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:38.006764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:38.007200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] PATH_INFO: `/v3/projects/9060b3b6ec4b4af883bc0f491fa4db29` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:38.010166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9060b3b6ec4b4af883bc0f491fa4db29)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:38.013428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:38.047038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:38.113791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9060b3b6ec4b4af883bc0f491fa4db29', 'request_id': 'req-fc7742de-9a42-4965-8b42-6fdc8c43bb24'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.123455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9060b3b6ec4b4af883bc0f491fa4db29', 'request_id': 'req-fc7742de-9a42-4965-8b42-6fdc8c43bb24'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.136216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9060b3b6ec4b4af883bc0f491fa4db29', 'request_id': 'req-fc7742de-9a42-4965-8b42-6fdc8c43bb24'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.152474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] The token cache is being invalidate because project 9060b3b6ec4b4af883bc0f491fa4db29 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:38.153573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.154467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.162893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 990/1989] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:37 2022] DELETE /identity/v3/projects/9060b3b6ec4b4af883bc0f491fa4db29 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:38.164241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c90ee68-0cec-48e5-8e6f-4a7ce555a553 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:38.182898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c90ee68-0cec-48e5-8e6f-4a7ce555a553 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:38.192412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:38.193822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:38.231553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:38.265640 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:38.266304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:38.266587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:38.266861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] PATH_INFO: `/v3/users/2e35f8dab49e483c963b1c5c124f05da` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:38.268078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e35f8dab49e483c963b1c5c124f05da)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:38.270597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:38.291796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc7742de-9a42-4965-8b42-6fdc8c43bb24 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:38.297857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:38.298656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:38.298940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:38.299216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] PATH_INFO: `/v3/users/b933c13f6d10469eacd1f748bd43ac11` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:38.300592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] RBAC: Authorizing `identity:delete_user(user_id=b933c13f6d10469eacd1f748bd43ac11)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:38.302737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:38.328882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e35f8dab49e483c963b1c5c124f05da'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.338940 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e35f8dab49e483c963b1c5c124f05da'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.339823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e35f8dab49e483c963b1c5c124f05da'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.340694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e35f8dab49e483c963b1c5c124f05da'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.349836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-26f0339b-7ed9-498e-863c-54a1a9cb4051 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e35f8dab49e483c963b1c5c124f05da'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.358211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b933c13f6d10469eacd1f748bd43ac11'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.358992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b933c13f6d10469eacd1f748bd43ac11'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.366269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b933c13f6d10469eacd1f748bd43ac11'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.367020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b933c13f6d10469eacd1f748bd43ac11'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.376133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b933c13f6d10469eacd1f748bd43ac11'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.376392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1000/1990] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:38 2022] DELETE /identity/v3/users/2e35f8dab49e483c963b1c5c124f05da => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:38.401157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 991/1991] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:38 2022] DELETE /identity/v3/users/b933c13f6d10469eacd1f748bd43ac11 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:38.405500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8a9c00-ccc7-4866-869e-fadbbb20fa5c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:38.406106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8a9c00-ccc7-4866-869e-fadbbb20fa5c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:38.406106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8a9c00-ccc7-4866-869e-fadbbb20fa5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:38.430133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7c8a9c00-ccc7-4866-869e-fadbbb20fa5c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:38.431352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7c8a9c00-ccc7-4866-869e-fadbbb20fa5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:38.437453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:38.439048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:38.475553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:38.526428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1001/1992] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:38.538946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8a9c00-ccc7-4866-869e-fadbbb20fa5c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:38.540862 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7c8a9c00-ccc7-4866-869e-fadbbb20fa5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:38.545662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee912ac3-fad6-4194-a535-7784563f7beb admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:38.556141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:38.558153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:38.558153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:38.558729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] PATH_INFO: `/v3/users/3572ca950d354407a707f9dd6383713e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:38.560808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] RBAC: Authorizing `identity:delete_user(user_id=3572ca950d354407a707f9dd6383713e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:38.563092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:38.634249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3572ca950d354407a707f9dd6383713e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.634249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3572ca950d354407a707f9dd6383713e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.646382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3572ca950d354407a707f9dd6383713e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.647317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8a9c00-ccc7-4866-869e-fadbbb20fa5c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:38.648239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3572ca950d354407a707f9dd6383713e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.655597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e822a320-7c9d-4bee-8e19-03433139e599 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:38.674777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e822a320-7c9d-4bee-8e19-03433139e599 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:38.675849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e822a320-7c9d-4bee-8e19-03433139e599 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:38.676242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0f66b525-8c48-4a64-8f08-e144aca0314c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3572ca950d354407a707f9dd6383713e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:38.676558 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e822a320-7c9d-4bee-8e19-03433139e599 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:38.679496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e822a320-7c9d-4bee-8e19-03433139e599 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:38.693341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:38.699146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e822a320-7c9d-4bee-8e19-03433139e599 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:38.701014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e822a320-7c9d-4bee-8e19-03433139e599 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:38.703126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 992/1993] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:38 2022] DELETE /identity/v3/users/3572ca950d354407a707f9dd6383713e => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:38.709118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d22a89-6406-4b7b-ac6f-63f07d999f9a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:38.709682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d22a89-6406-4b7b-ac6f-63f07d999f9a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:38.711022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d22a89-6406-4b7b-ac6f-63f07d999f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:38.714228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1002/1994] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:38.735241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-69d22a89-6406-4b7b-ac6f-63f07d999f9a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:38.739182 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-69d22a89-6406-4b7b-ac6f-63f07d999f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:38.820948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 993/1995] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:38.834324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e822a320-7c9d-4bee-8e19-03433139e599 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:38.836179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e822a320-7c9d-4bee-8e19-03433139e599 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:38.850614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e822a320-7c9d-4bee-8e19-03433139e599 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:38.907430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d22a89-6406-4b7b-ac6f-63f07d999f9a None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:38.908866 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-69d22a89-6406-4b7b-ac6f-63f07d999f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:38.926423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d22a89-6406-4b7b-ac6f-63f07d999f9a None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:38.936023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:38.937335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:38.938400 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:38.939032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] PATH_INFO: `/v3/projects/643e63a9e0154da1bc4f0f2d1c5c7f56` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:38.945232 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] RBAC: Authorizing `identity:delete_project(project_id=643e63a9e0154da1bc4f0f2d1c5c7f56)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:38.950088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:38.969841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ca5e8c-91c8-4290-b631-a3018f024026 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:38.970465 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ca5e8c-91c8-4290-b631-a3018f024026 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:38.970784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ca5e8c-91c8-4290-b631-a3018f024026 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:38.971126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ca5e8c-91c8-4290-b631-a3018f024026 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:38.985758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ca5e8c-91c8-4290-b631-a3018f024026 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:38.988196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ca5e8c-91c8-4290-b631-a3018f024026 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:39.000222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1003/1996] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:39.014541 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:39.038254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '643e63a9e0154da1bc4f0f2d1c5c7f56', 'request_id': 'req-0195b130-413d-4080-a11a-61cead1f9c26'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.047983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '643e63a9e0154da1bc4f0f2d1c5c7f56', 'request_id': 'req-0195b130-413d-4080-a11a-61cead1f9c26'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.055754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '643e63a9e0154da1bc4f0f2d1c5c7f56', 'request_id': 'req-0195b130-413d-4080-a11a-61cead1f9c26'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.068298 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] The token cache is being invalidate because project 643e63a9e0154da1bc4f0f2d1c5c7f56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:39.068566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.069193 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-0195b130-413d-4080-a11a-61cead1f9c26 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.070732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 994/1997] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:38 2022] DELETE /identity/v3/projects/643e63a9e0154da1bc4f0f2d1c5c7f56 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:39.075729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638f6f4a-b3bf-4831-be7c-31b443ffe33a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.076401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638f6f4a-b3bf-4831-be7c-31b443ffe33a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.076949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638f6f4a-b3bf-4831-be7c-31b443ffe33a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.100612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-638f6f4a-b3bf-4831-be7c-31b443ffe33a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:39.102897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-638f6f4a-b3bf-4831-be7c-31b443ffe33a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:39.181755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1004/1998] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:39.191513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd34196b-8506-485e-acca-35368353a45d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.195991 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd34196b-8506-485e-acca-35368353a45d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.196299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd34196b-8506-485e-acca-35368353a45d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.227752 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-bd34196b-8506-485e-acca-35368353a45d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:39.234843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638f6f4a-b3bf-4831-be7c-31b443ffe33a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:39.235612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-bd34196b-8506-485e-acca-35368353a45d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:39.238785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-638f6f4a-b3bf-4831-be7c-31b443ffe33a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:39.254416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638f6f4a-b3bf-4831-be7c-31b443ffe33a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:39.264870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 995/1999] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:39.267235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd34196b-8506-485e-acca-35368353a45d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:39.274644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:39.275298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.275570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.275838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] PATH_INFO: `/v3/projects/ea835a465ba545079d6b34e856015592` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.279421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea835a465ba545079d6b34e856015592)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:39.286447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:39.291508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd34196b-8506-485e-acca-35368353a45d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:39.301349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cfb809-3e23-4516-8c05-721ff571a1ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:39.302033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cfb809-3e23-4516-8c05-721ff571a1ba admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.302346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cfb809-3e23-4516-8c05-721ff571a1ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.302666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cfb809-3e23-4516-8c05-721ff571a1ba admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.303377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cfb809-3e23-4516-8c05-721ff571a1ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:39.303732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cfb809-3e23-4516-8c05-721ff571a1ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:39.305509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cfb809-3e23-4516-8c05-721ff571a1ba admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:39.324034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 996/2000] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:39.324958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cfb809-3e23-4516-8c05-721ff571a1ba admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:39.335816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cfb809-3e23-4516-8c05-721ff571a1ba admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:39.341218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d728f5fb-a44d-471d-9651-879430bf1bac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:39.341834 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d728f5fb-a44d-471d-9651-879430bf1bac admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.342175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d728f5fb-a44d-471d-9651-879430bf1bac admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.342452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d728f5fb-a44d-471d-9651-879430bf1bac admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.343324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d728f5fb-a44d-471d-9651-879430bf1bac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:39.347362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d728f5fb-a44d-471d-9651-879430bf1bac admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:39.384184 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 997/2001] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:39 2022] POST /identity/v3/projects => generated 409 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:39.387419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea835a465ba545079d6b34e856015592', 'request_id': 'req-4fb30879-5010-45f2-8e3a-1564c303ce1a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.389005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d728f5fb-a44d-471d-9651-879430bf1bac admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:39.400088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea835a465ba545079d6b34e856015592', 'request_id': 'req-4fb30879-5010-45f2-8e3a-1564c303ce1a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.416821 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea835a465ba545079d6b34e856015592', 'request_id': 'req-4fb30879-5010-45f2-8e3a-1564c303ce1a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.434742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] The token cache is being invalidate because project ea835a465ba545079d6b34e856015592 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:39.435125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.435800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.440942 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1005/2002] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:39 2022] DELETE /identity/v3/projects/ea835a465ba545079d6b34e856015592 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:39.441925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:39.443429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:39.498757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d728f5fb-a44d-471d-9651-879430bf1bac admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:39.502282 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:39.509092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382defa-2233-447a-b600-1acf2dde7fef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:39.510911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382defa-2233-447a-b600-1acf2dde7fef admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.511262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382defa-2233-447a-b600-1acf2dde7fef admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.511957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2382defa-2233-447a-b600-1acf2dde7fef admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.512784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2382defa-2233-447a-b600-1acf2dde7fef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:39.515082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2382defa-2233-447a-b600-1acf2dde7fef admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:39.548304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 998/2003] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:39 2022] POST /identity/v3/users => generated 305 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:39.549583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382defa-2233-447a-b600-1acf2dde7fef admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:39.552315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2382defa-2233-447a-b600-1acf2dde7fef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:39.566901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2382defa-2233-447a-b600-1acf2dde7fef admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:39.571881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:39.572454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.572733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.573011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] PATH_INFO: `/v3/projects/19744acd950444688b96495eba73039a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.574147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19744acd950444688b96495eba73039a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:39.575338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb30879-5010-45f2-8e3a-1564c303ce1a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:39.576100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:39.581548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:39.582222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.582610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.582765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] PATH_INFO: `/v3/users/7444475ae92442979967ec34d69a8c28` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.583905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7444475ae92442979967ec34d69a8c28)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:39.585988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:39.640267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19744acd950444688b96495eba73039a', 'request_id': 'req-d8c810dc-74ba-442f-a390-21df2c7a5580'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.644497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7444475ae92442979967ec34d69a8c28'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.651101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19744acd950444688b96495eba73039a', 'request_id': 'req-d8c810dc-74ba-442f-a390-21df2c7a5580'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.652860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7444475ae92442979967ec34d69a8c28'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.653470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7444475ae92442979967ec34d69a8c28'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.654447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7444475ae92442979967ec34d69a8c28'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.661136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19744acd950444688b96495eba73039a', 'request_id': 'req-d8c810dc-74ba-442f-a390-21df2c7a5580'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.667563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:DELETE=11,keystone:SELECT=83,keystone:INSERT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:39.673467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7444475ae92442979967ec34d69a8c28'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.679196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] The token cache is being invalidate because project 19744acd950444688b96495eba73039a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:39.679914 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.681384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.683867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 999/2004] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:39 2022] DELETE /identity/v3/projects/19744acd950444688b96495eba73039a => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:39.688805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:39.696504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:39.707826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1006/2005] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:39 2022] DELETE /identity/v3/users/7444475ae92442979967ec34d69a8c28 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:39.709110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:39.723663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:39.795698 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c810dc-74ba-442f-a390-21df2c7a5580 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:39.840542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4b640b-441f-4353-a4ed-668abf617d63 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:39.854501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:39.857492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.857755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.858092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] PATH_INFO: `/v3/users/d92f520659d94fce87044fb2f33a67bf` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.859482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d92f520659d94fce87044fb2f33a67bf)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:39.861979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:39.887097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a16bc65-7c8c-43f3-9086-e5b35a770e37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:39.888156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a16bc65-7c8c-43f3-9086-e5b35a770e37 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.888699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a16bc65-7c8c-43f3-9086-e5b35a770e37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.889213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a16bc65-7c8c-43f3-9086-e5b35a770e37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.890992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a16bc65-7c8c-43f3-9086-e5b35a770e37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:39.893463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a16bc65-7c8c-43f3-9086-e5b35a770e37 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:39.924373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:39.928616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1000/2006] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:39 2022] GET /identity/v3/roles => generated 1420 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:39.933850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a16bc65-7c8c-43f3-9086-e5b35a770e37 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:39.938146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd92f520659d94fce87044fb2f33a67bf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.948050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd92f520659d94fce87044fb2f33a67bf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.948938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd92f520659d94fce87044fb2f33a67bf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.949900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd92f520659d94fce87044fb2f33a67bf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.962011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a16bc65-7c8c-43f3-9086-e5b35a770e37 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:39.964418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-dc4bb30b-b3b6-40ce-b975-ccd01e4ffd07 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd92f520659d94fce87044fb2f33a67bf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:39.969485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd26acd-0d09-4239-bbc7-5c95a87a3f18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:39.970226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd26acd-0d09-4239-bbc7-5c95a87a3f18 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:39.970665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd26acd-0d09-4239-bbc7-5c95a87a3f18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:39.971010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd26acd-0d09-4239-bbc7-5c95a87a3f18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:39.971913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd26acd-0d09-4239-bbc7-5c95a87a3f18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:39.974506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd26acd-0d09-4239-bbc7-5c95a87a3f18 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:39.989821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1001/2007] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:39 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:39.990889 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd26acd-0d09-4239-bbc7-5c95a87a3f18 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:39.995003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3bd26acd-0d09-4239-bbc7-5c95a87a3f18 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:39.999759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1007/2008] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:39 2022] DELETE /identity/v3/users/d92f520659d94fce87044fb2f33a67bf => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:40.009862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd26acd-0d09-4239-bbc7-5c95a87a3f18 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:40.083201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0003d09a-3adf-465b-b987-4239e74d18f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:40.083991 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0003d09a-3adf-465b-b987-4239e74d18f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.084273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0003d09a-3adf-465b-b987-4239e74d18f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.084548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0003d09a-3adf-465b-b987-4239e74d18f3 admin admin] PATH_INFO: `/v3/projects/4cefe7c3f4a84ba1952df6e210278d78/users/00bc0881a0ce49d294f480d7d7bb0488/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.101184 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0003d09a-3adf-465b-b987-4239e74d18f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4cefe7c3f4a84ba1952df6e210278d78, user_id=00bc0881a0ce49d294f480d7d7bb0488, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:40.104138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0003d09a-3adf-465b-b987-4239e74d18f3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:40.120636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1002/2009] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:39 2022] PUT /identity/v3/projects/4cefe7c3f4a84ba1952df6e210278d78/users/00bc0881a0ce49d294f480d7d7bb0488/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:40.132724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1a66c8-9bb4-47d0-964d-1b3730a83321 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.133007 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1a66c8-9bb4-47d0-964d-1b3730a83321 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.133281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1a66c8-9bb4-47d0-964d-1b3730a83321 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.153825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0003d09a-3adf-465b-b987-4239e74d18f3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.155617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0003d09a-3adf-465b-b987-4239e74d18f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:40.168870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:DELETE=1,keystone:SELECT=98,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:40.176072 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5a1a66c8-9bb4-47d0-964d-1b3730a83321 None None] MFA Rules not processed for user `00bc0881a0ce49d294f480d7d7bb0488`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:40.178212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5a1a66c8-9bb4-47d0-964d-1b3730a83321 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:40.242775 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0003d09a-3adf-465b-b987-4239e74d18f3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:40.248243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:40.248952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.249302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.249672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] PATH_INFO: `/v3/projects/6db6da59f5894fdeaf68de61254b28f6` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.250897 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6db6da59f5894fdeaf68de61254b28f6)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:40.252916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:40.273671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1008/2010] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:40 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:40.278334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1a66c8-9bb4-47d0-964d-1b3730a83321 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.292238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1a66c8-9bb4-47d0-964d-1b3730a83321 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:40.310755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6db6da59f5894fdeaf68de61254b28f6', 'request_id': 'req-73fc67e3-416b-4a88-84fa-ec52769448a6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.320720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6db6da59f5894fdeaf68de61254b28f6', 'request_id': 'req-73fc67e3-416b-4a88-84fa-ec52769448a6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.328987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6db6da59f5894fdeaf68de61254b28f6', 'request_id': 'req-73fc67e3-416b-4a88-84fa-ec52769448a6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.345435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] The token cache is being invalidate because project 6db6da59f5894fdeaf68de61254b28f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:40.345719 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.346565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.352165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1003/2011] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:40 2022] DELETE /identity/v3/projects/6db6da59f5894fdeaf68de61254b28f6 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:40.352165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.352165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:40.360495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f753b49-8452-4c99-a803-85a826fd5276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:40.361077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f753b49-8452-4c99-a803-85a826fd5276 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.361354 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f753b49-8452-4c99-a803-85a826fd5276 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.361747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f753b49-8452-4c99-a803-85a826fd5276 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.363017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f753b49-8452-4c99-a803-85a826fd5276 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:40.364912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f753b49-8452-4c99-a803-85a826fd5276 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:40.394341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1009/2012] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:40 2022] POST /identity/v3/projects => generated 409 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:40.395299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f753b49-8452-4c99-a803-85a826fd5276 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.397251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4f753b49-8452-4c99-a803-85a826fd5276 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:40.419312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73fc67e3-416b-4a88-84fa-ec52769448a6 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:40.433515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:40.463075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:40.465234 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f753b49-8452-4c99-a803-85a826fd5276 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:40.470311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:40.470990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.471513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.472145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] PATH_INFO: `/v3/users/8072e00f0d764d04a92e635e43f694a2` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.473648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8072e00f0d764d04a92e635e43f694a2)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:40.475767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:40.477735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:40.478395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.478669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.478964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] PATH_INFO: `/v3/projects/bef8c130f3284910ad0c635ce534b48d` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.480061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bef8c130f3284910ad0c635ce534b48d)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:40.483279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:40.525014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8072e00f0d764d04a92e635e43f694a2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.529646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bef8c130f3284910ad0c635ce534b48d', 'request_id': 'req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.532025 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8072e00f0d764d04a92e635e43f694a2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.532699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8072e00f0d764d04a92e635e43f694a2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.533321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8072e00f0d764d04a92e635e43f694a2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.537994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bef8c130f3284910ad0c635ce534b48d', 'request_id': 'req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.541916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bdd873bd-e834-43ce-912a-f949b9c199a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8072e00f0d764d04a92e635e43f694a2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.545316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bef8c130f3284910ad0c635ce534b48d', 'request_id': 'req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.555937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] The token cache is being invalidate because project bef8c130f3284910ad0c635ce534b48d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:40.556224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.556870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:40.558427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1004/2013] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:40 2022] DELETE /identity/v3/projects/bef8c130f3284910ad0c635ce534b48d => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:40.559161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.560435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:40.563188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1010/2014] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:40 2022] DELETE /identity/v3/users/8072e00f0d764d04a92e635e43f694a2 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:40.564202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881bb099-9496-415d-a3a5-cfee63de2ad7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.564604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881bb099-9496-415d-a3a5-cfee63de2ad7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.564875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881bb099-9496-415d-a3a5-cfee63de2ad7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.586548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-881bb099-9496-415d-a3a5-cfee63de2ad7 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:40.587830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-881bb099-9496-415d-a3a5-cfee63de2ad7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:40.679754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1011/2015] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:40.685249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f4d9b7-7b4b-4660-be75-dcc50b5bd1a8 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:40.689873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed9bde0-56cf-4b72-b86e-bd68d52576f2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.690012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed9bde0-56cf-4b72-b86e-bd68d52576f2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.690298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed9bde0-56cf-4b72-b86e-bd68d52576f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.690986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e92c3be-409c-45df-9c83-4d63a954d44e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:40.691753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e92c3be-409c-45df-9c83-4d63a954d44e admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.692148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e92c3be-409c-45df-9c83-4d63a954d44e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.692534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e92c3be-409c-45df-9c83-4d63a954d44e admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.693360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e92c3be-409c-45df-9c83-4d63a954d44e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:40.695708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e92c3be-409c-45df-9c83-4d63a954d44e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:40.720155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7ed9bde0-56cf-4b72-b86e-bd68d52576f2 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:40.721376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7ed9bde0-56cf-4b72-b86e-bd68d52576f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:40.725316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1005/2016] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:40 2022] POST /identity/v3/users => generated 311 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:40.738552 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1012/2017] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:40.743033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed9bde0-56cf-4b72-b86e-bd68d52576f2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.745266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e92c3be-409c-45df-9c83-4d63a954d44e admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.755311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed9bde0-56cf-4b72-b86e-bd68d52576f2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:40.760755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd0bc30-703d-40f8-b03d-4fd3799abedd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:40.761489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd0bc30-703d-40f8-b03d-4fd3799abedd admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.761859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd0bc30-703d-40f8-b03d-4fd3799abedd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.762253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd0bc30-703d-40f8-b03d-4fd3799abedd admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.762998 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e92c3be-409c-45df-9c83-4d63a954d44e admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:40.763120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd0bc30-703d-40f8-b03d-4fd3799abedd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:40.763522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd0bc30-703d-40f8-b03d-4fd3799abedd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:40.765288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd0bc30-703d-40f8-b03d-4fd3799abedd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:40.769334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fb9a7c-e67a-4320-af1e-3a9bfdce17a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:40.769951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fb9a7c-e67a-4320-af1e-3a9bfdce17a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.770350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fb9a7c-e67a-4320-af1e-3a9bfdce17a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.770622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01fb9a7c-e67a-4320-af1e-3a9bfdce17a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.771262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01fb9a7c-e67a-4320-af1e-3a9bfdce17a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:40.772823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01fb9a7c-e67a-4320-af1e-3a9bfdce17a1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:40.784660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1013/2018] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:40.790833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1006/2019] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:40 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:40.795397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd0bc30-703d-40f8-b03d-4fd3799abedd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.796174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fb9a7c-e67a-4320-af1e-3a9bfdce17a1 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.807174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd0bc30-703d-40f8-b03d-4fd3799abedd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:40.808498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01fb9a7c-e67a-4320-af1e-3a9bfdce17a1 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:40.813997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4b657e-d450-4aad-8439-0aa6ebd3eb50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:40.814239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4b657e-d450-4aad-8439-0aa6ebd3eb50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.814347 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4b657e-d450-4aad-8439-0aa6ebd3eb50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.814701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4b657e-d450-4aad-8439-0aa6ebd3eb50 admin admin] PATH_INFO: `/v3/projects/b1414ab312fe47e88c718e1b8d6bfb04/users/631464266d86495aa9e0ad85a32a80fb/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.818469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a11362-12e4-44ff-8145-2ef113ad0564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:40.819030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a11362-12e4-44ff-8145-2ef113ad0564 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:40.819406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a11362-12e4-44ff-8145-2ef113ad0564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:40.819773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a11362-12e4-44ff-8145-2ef113ad0564 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:40.820508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a11362-12e4-44ff-8145-2ef113ad0564 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:40.822591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a11362-12e4-44ff-8145-2ef113ad0564 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:40.835571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4b657e-d450-4aad-8439-0aa6ebd3eb50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1414ab312fe47e88c718e1b8d6bfb04, user_id=631464266d86495aa9e0ad85a32a80fb, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:40.840511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4b657e-d450-4aad-8439-0aa6ebd3eb50 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:40.862098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1007/2020] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:40 2022] PUT /identity/v3/projects/b1414ab312fe47e88c718e1b8d6bfb04/users/631464266d86495aa9e0ad85a32a80fb/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:40.866627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1014/2021] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:40 2022] POST /identity/v3/projects => generated 363 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:40.867532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a11362-12e4-44ff-8145-2ef113ad0564 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.869514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4b657e-d450-4aad-8439-0aa6ebd3eb50 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:40.884528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:40.946104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:40.979878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a11362-12e4-44ff-8145-2ef113ad0564 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.002217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a435ef-79f2-45df-afcf-78725e6ee07e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.003081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a435ef-79f2-45df-afcf-78725e6ee07e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.003373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a435ef-79f2-45df-afcf-78725e6ee07e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.003621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a435ef-79f2-45df-afcf-78725e6ee07e admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.004318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a435ef-79f2-45df-afcf-78725e6ee07e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.008042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a435ef-79f2-45df-afcf-78725e6ee07e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.042513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4b657e-d450-4aad-8439-0aa6ebd3eb50 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.047570 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b456249d-5a79-490a-a77c-376bb99129d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.048191 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b456249d-5a79-490a-a77c-376bb99129d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.048471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b456249d-5a79-490a-a77c-376bb99129d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.048746 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b456249d-5a79-490a-a77c-376bb99129d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.049394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b456249d-5a79-490a-a77c-376bb99129d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.051605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b456249d-5a79-490a-a77c-376bb99129d7 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.054409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1015/2022] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:40 2022] POST /identity/v3/users => generated 282 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:41.059047 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a435ef-79f2-45df-afcf-78725e6ee07e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.062974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-07a435ef-79f2-45df-afcf-78725e6ee07e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:41.067747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1008/2023] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:40 2022] GET /identity/v3/roles => generated 1420 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:41.067747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b456249d-5a79-490a-a77c-376bb99129d7 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.076367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b456249d-5a79-490a-a77c-376bb99129d7 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.076940 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a435ef-79f2-45df-afcf-78725e6ee07e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.081303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6316ee15-7532-4cd9-bece-8c6769b55c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.082219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6316ee15-7532-4cd9-bece-8c6769b55c44 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.082504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6316ee15-7532-4cd9-bece-8c6769b55c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.082783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6316ee15-7532-4cd9-bece-8c6769b55c44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.083352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.083490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6316ee15-7532-4cd9-bece-8c6769b55c44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.084151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.084538 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.084951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] PATH_INFO: `/v3/projects/8d59bc1a29c049368c9126b5a63cd685` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.085269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6316ee15-7532-4cd9-bece-8c6769b55c44 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.087378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d59bc1a29c049368c9126b5a63cd685)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.090290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.098109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1009/2024] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:41.099056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6316ee15-7532-4cd9-bece-8c6769b55c44 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.110362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6316ee15-7532-4cd9-bece-8c6769b55c44 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.115804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d89d82-4fb5-4c68-bcea-4d359b56b3b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.116601 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d89d82-4fb5-4c68-bcea-4d359b56b3b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.116737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d89d82-4fb5-4c68-bcea-4d359b56b3b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.117018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d89d82-4fb5-4c68-bcea-4d359b56b3b6 admin admin] PATH_INFO: `/v3/projects/b1414ab312fe47e88c718e1b8d6bfb04/users/631464266d86495aa9e0ad85a32a80fb/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.119116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d89d82-4fb5-4c68-bcea-4d359b56b3b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1414ab312fe47e88c718e1b8d6bfb04, user_id=631464266d86495aa9e0ad85a32a80fb, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.122021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d89d82-4fb5-4c68-bcea-4d359b56b3b6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.138974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1010/2025] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:41 2022] PUT /identity/v3/projects/b1414ab312fe47e88c718e1b8d6bfb04/users/631464266d86495aa9e0ad85a32a80fb/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:41.139992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d89d82-4fb5-4c68-bcea-4d359b56b3b6 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.170302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d59bc1a29c049368c9126b5a63cd685', 'request_id': 'req-3bed6cc5-491e-462b-9aa4-05d86bad6454'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:41.179475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d59bc1a29c049368c9126b5a63cd685', 'request_id': 'req-3bed6cc5-491e-462b-9aa4-05d86bad6454'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:41.203334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d59bc1a29c049368c9126b5a63cd685', 'request_id': 'req-3bed6cc5-491e-462b-9aa4-05d86bad6454'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:41.210714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d89d82-4fb5-4c68-bcea-4d359b56b3b6 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.214193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] The token cache is being invalidate because project 8d59bc1a29c049368c9126b5a63cd685 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:41.214481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:41.215148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3bed6cc5-491e-462b-9aa4-05d86bad6454 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:41.216674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1016/2026] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:41 2022] DELETE /identity/v3/projects/8d59bc1a29c049368c9126b5a63cd685 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:41.217987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfdf635-3ff2-4c07-baf2-e36e9bf21805 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.218299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfdf635-3ff2-4c07-baf2-e36e9bf21805 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.218618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfdf635-3ff2-4c07-baf2-e36e9bf21805 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.246299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cdfdf635-3ff2-4c07-baf2-e36e9bf21805 None None] MFA Rules not processed for user `631464266d86495aa9e0ad85a32a80fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:41.247575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cdfdf635-3ff2-4c07-baf2-e36e9bf21805 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:41.272424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352b2e78-4df8-4809-b2e9-7ca815e0bac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.273036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352b2e78-4df8-4809-b2e9-7ca815e0bac2 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.273336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352b2e78-4df8-4809-b2e9-7ca815e0bac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.273606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352b2e78-4df8-4809-b2e9-7ca815e0bac2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.274269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352b2e78-4df8-4809-b2e9-7ca815e0bac2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.275882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352b2e78-4df8-4809-b2e9-7ca815e0bac2 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.286048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1011/2027] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:41.289370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff1e30-7e34-402b-a14a-6dda87cad17f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.289370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff1e30-7e34-402b-a14a-6dda87cad17f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.289370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ff1e30-7e34-402b-a14a-6dda87cad17f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.310194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-29ff1e30-7e34-402b-a14a-6dda87cad17f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:41.312177 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-29ff1e30-7e34-402b-a14a-6dda87cad17f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:41.321233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1012/2028] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:41.322010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff1e30-7e34-402b-a14a-6dda87cad17f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.326563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1017/2029] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:41 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:41.327028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-29ff1e30-7e34-402b-a14a-6dda87cad17f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:41.329683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e41611-476c-4054-a136-7b3f1b86c699 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.330175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e41611-476c-4054-a136-7b3f1b86c699 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.330454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e41611-476c-4054-a136-7b3f1b86c699 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.335336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ff1e30-7e34-402b-a14a-6dda87cad17f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.340060 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26bd5d5-e11a-45c3-afd6-cc16000715a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.340842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26bd5d5-e11a-45c3-afd6-cc16000715a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.341268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26bd5d5-e11a-45c3-afd6-cc16000715a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.341631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26bd5d5-e11a-45c3-afd6-cc16000715a7 admin admin] PATH_INFO: `/v3/projects/ce60d6a8cea344a3b7d2e332fcfeb8fc/users/0702c01019ee4a358b9243c0e38cdc20/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.350193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e7e41611-476c-4054-a136-7b3f1b86c699 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:41.351464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e7e41611-476c-4054-a136-7b3f1b86c699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:41.357179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26bd5d5-e11a-45c3-afd6-cc16000715a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce60d6a8cea344a3b7d2e332fcfeb8fc, user_id=0702c01019ee4a358b9243c0e38cdc20, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.359830 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26bd5d5-e11a-45c3-afd6-cc16000715a7 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.360433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1018/2030] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:41.363437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e41611-476c-4054-a136-7b3f1b86c699 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.364964 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e7e41611-476c-4054-a136-7b3f1b86c699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:41.371100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:41.374900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1013/2031] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:41 2022] PUT /identity/v3/projects/ce60d6a8cea344a3b7d2e332fcfeb8fc/users/0702c01019ee4a358b9243c0e38cdc20/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:41.375719 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26bd5d5-e11a-45c3-afd6-cc16000715a7 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.400304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:41.432653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e41611-476c-4054-a136-7b3f1b86c699 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.449250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26bd5d5-e11a-45c3-afd6-cc16000715a7 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.454466 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af3a9b6-5a8d-4ec5-8f0f-1fea9398bbf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.455091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af3a9b6-5a8d-4ec5-8f0f-1fea9398bbf9 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.455411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af3a9b6-5a8d-4ec5-8f0f-1fea9398bbf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.455648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af3a9b6-5a8d-4ec5-8f0f-1fea9398bbf9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.456824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af3a9b6-5a8d-4ec5-8f0f-1fea9398bbf9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:41.457128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af3a9b6-5a8d-4ec5-8f0f-1fea9398bbf9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.458768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af3a9b6-5a8d-4ec5-8f0f-1fea9398bbf9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.471501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1014/2032] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:41.476120 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8abf9c-cf40-4aa9-b1f8-f1e9ea5957a0 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.476407 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8abf9c-cf40-4aa9-b1f8-f1e9ea5957a0 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.476686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8abf9c-cf40-4aa9-b1f8-f1e9ea5957a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.484811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe18217-4ed1-4424-92dc-60f2676e19c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.486328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe18217-4ed1-4424-92dc-60f2676e19c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.486631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe18217-4ed1-4424-92dc-60f2676e19c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.486944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe18217-4ed1-4424-92dc-60f2676e19c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.508440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4e8abf9c-cf40-4aa9-b1f8-f1e9ea5957a0 None None] MFA Rules not processed for user `0702c01019ee4a358b9243c0e38cdc20`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:41.509659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4e8abf9c-cf40-4aa9-b1f8-f1e9ea5957a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:41.552576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe18217-4ed1-4424-92dc-60f2676e19c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.554019 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe18217-4ed1-4424-92dc-60f2676e19c4 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.563721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1019/2033] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:41 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:41.564509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe18217-4ed1-4424-92dc-60f2676e19c4 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.575113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe18217-4ed1-4424-92dc-60f2676e19c4 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.580282 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1015/2034] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:41 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:41.581953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce32b16-6721-4d9c-a1b1-7626913a118a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.582757 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce32b16-6721-4d9c-a1b1-7626913a118a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.583029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce32b16-6721-4d9c-a1b1-7626913a118a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.583301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce32b16-6721-4d9c-a1b1-7626913a118a admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.583931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce32b16-6721-4d9c-a1b1-7626913a118a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.586467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce32b16-6721-4d9c-a1b1-7626913a118a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.587076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8abf9c-cf40-4aa9-b1f8-f1e9ea5957a0 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.602598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8abf9c-cf40-4aa9-b1f8-f1e9ea5957a0 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.609071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b6edfd-1d01-40f5-910e-d763de8b43b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.613241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6edfd-1d01-40f5-910e-d763de8b43b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.613688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6edfd-1d01-40f5-910e-d763de8b43b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.614104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6edfd-1d01-40f5-910e-d763de8b43b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.614885 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b6edfd-1d01-40f5-910e-d763de8b43b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.617178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b6edfd-1d01-40f5-910e-d763de8b43b2 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.621930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1020/2035] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:41 2022] POST /identity/v3/projects => generated 385 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:41.625753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce32b16-6721-4d9c-a1b1-7626913a118a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.669274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1016/2036] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:41 2022] POST /identity/v3/projects => generated 363 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:41.673843 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b6edfd-1d01-40f5-910e-d763de8b43b2 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.687018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:41.706979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce32b16-6721-4d9c-a1b1-7626913a118a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.751990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b6edfd-1d01-40f5-910e-d763de8b43b2 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.757370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9278024-af17-420f-9c60-5b9f17810afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.757979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9278024-af17-420f-9c60-5b9f17810afc admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.758331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9278024-af17-420f-9c60-5b9f17810afc admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.758675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9278024-af17-420f-9c60-5b9f17810afc admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.759257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9278024-af17-420f-9c60-5b9f17810afc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.761940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9278024-af17-420f-9c60-5b9f17810afc admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.790974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1d677a-f8a0-42fb-94f7-f7efe94a7d77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.791613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1d677a-f8a0-42fb-94f7-f7efe94a7d77 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.791925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1d677a-f8a0-42fb-94f7-f7efe94a7d77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.792201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1d677a-f8a0-42fb-94f7-f7efe94a7d77 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.792869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1d677a-f8a0-42fb-94f7-f7efe94a7d77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.794831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1d677a-f8a0-42fb-94f7-f7efe94a7d77 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.799336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1017/2037] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:41 2022] POST /identity/v3/users => generated 288 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:41.811760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9278024-af17-420f-9c60-5b9f17810afc admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.814040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e9278024-af17-420f-9c60-5b9f17810afc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:41.825712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:41.833101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1021/2038] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:41 2022] POST /identity/v3/users => generated 293 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:41.833944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1d677a-f8a0-42fb-94f7-f7efe94a7d77 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.856883 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1d677a-f8a0-42fb-94f7-f7efe94a7d77 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.863040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e700e9-d157-46e6-9931-99e84ba14f57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.864013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e700e9-d157-46e6-9931-99e84ba14f57 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.864013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e700e9-d157-46e6-9931-99e84ba14f57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.864243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e700e9-d157-46e6-9931-99e84ba14f57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.864817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e700e9-d157-46e6-9931-99e84ba14f57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.866859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e700e9-d157-46e6-9931-99e84ba14f57 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.880824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1022/2039] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:41.880824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e700e9-d157-46e6-9931-99e84ba14f57 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.889246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e700e9-d157-46e6-9931-99e84ba14f57 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.894461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6115621-b860-4301-9ee9-4e793e9ec886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.895065 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6115621-b860-4301-9ee9-4e793e9ec886 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.895776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6115621-b860-4301-9ee9-4e793e9ec886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.895776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6115621-b860-4301-9ee9-4e793e9ec886 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.896369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6115621-b860-4301-9ee9-4e793e9ec886 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.898258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6115621-b860-4301-9ee9-4e793e9ec886 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.911003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1023/2040] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:41.911892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6115621-b860-4301-9ee9-4e793e9ec886 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:41.916940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9278024-af17-420f-9c60-5b9f17810afc admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.924426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38080784-5cc3-487d-bd98-912a99c451ea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.925308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38080784-5cc3-487d-bd98-912a99c451ea service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.925421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38080784-5cc3-487d-bd98-912a99c451ea service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.925936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38080784-5cc3-487d-bd98-912a99c451ea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.927552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-38080784-5cc3-487d-bd98-912a99c451ea service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:41.931335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6115621-b860-4301-9ee9-4e793e9ec886 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:41.948004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57697434-9f2f-4616-9b94-f3620ca2e09b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:41.949321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57697434-9f2f-4616-9b94-f3620ca2e09b admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:41.949601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57697434-9f2f-4616-9b94-f3620ca2e09b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:41.949926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57697434-9f2f-4616-9b94-f3620ca2e09b admin admin] PATH_INFO: `/v3/projects/b1b894867e354880b1fda81bebdbeae1/users/c96ad23928314af28c8e820883c60a55/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:41.970989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57697434-9f2f-4616-9b94-f3620ca2e09b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1b894867e354880b1fda81bebdbeae1, user_id=c96ad23928314af28c8e820883c60a55, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:41.974596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57697434-9f2f-4616-9b94-f3620ca2e09b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:41.993918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1024/2041] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:41 2022] PUT /identity/v3/projects/b1b894867e354880b1fda81bebdbeae1/users/c96ad23928314af28c8e820883c60a55/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:41.997977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57697434-9f2f-4616-9b94-f3620ca2e09b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.008171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38080784-5cc3-487d-bd98-912a99c451ea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.009644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38080784-5cc3-487d-bd98-912a99c451ea service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.020380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1018/2042] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:41 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:42.021443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38080784-5cc3-487d-bd98-912a99c451ea service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.052811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38080784-5cc3-487d-bd98-912a99c451ea service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:42.102941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57697434-9f2f-4616-9b94-f3620ca2e09b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:42.119914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c0a826-7f01-42e6-a172-1092a46fc876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:42.120629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c0a826-7f01-42e6-a172-1092a46fc876 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.121253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c0a826-7f01-42e6-a172-1092a46fc876 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.121611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c0a826-7f01-42e6-a172-1092a46fc876 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.122377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c0a826-7f01-42e6-a172-1092a46fc876 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.124049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c0a826-7f01-42e6-a172-1092a46fc876 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.136156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1025/2043] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:41 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:42.140690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c0a826-7f01-42e6-a172-1092a46fc876 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.143140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6749c07-f97a-4094-9e74-26bace95047d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:42.143798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6749c07-f97a-4094-9e74-26bace95047d admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.144076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6749c07-f97a-4094-9e74-26bace95047d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.144354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6749c07-f97a-4094-9e74-26bace95047d admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.145090 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6749c07-f97a-4094-9e74-26bace95047d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.147805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6749c07-f97a-4094-9e74-26bace95047d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.152272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c0a826-7f01-42e6-a172-1092a46fc876 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:42.157970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f275173-f7d8-40a2-b939-a4e15fabceaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:42.158756 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f275173-f7d8-40a2-b939-a4e15fabceaa admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.159294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f275173-f7d8-40a2-b939-a4e15fabceaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.162594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1019/2044] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:42 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:42.162752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f275173-f7d8-40a2-b939-a4e15fabceaa admin admin] PATH_INFO: `/v3/projects/c2da67bdb084426ea652d7ff328c42bb/users/29f5e2c344cc4b17be1b175e934a9245/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.164053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6749c07-f97a-4094-9e74-26bace95047d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.176389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6749c07-f97a-4094-9e74-26bace95047d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:42.180993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f275173-f7d8-40a2-b939-a4e15fabceaa admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2da67bdb084426ea652d7ff328c42bb, user_id=29f5e2c344cc4b17be1b175e934a9245, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.183620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a240ec-345d-4071-82d2-554eff2fe043 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:42.184345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f275173-f7d8-40a2-b939-a4e15fabceaa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.184596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a240ec-345d-4071-82d2-554eff2fe043 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.184947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a240ec-345d-4071-82d2-554eff2fe043 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.185296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a240ec-345d-4071-82d2-554eff2fe043 admin admin] PATH_INFO: `/v3/projects/b1b894867e354880b1fda81bebdbeae1/users/c96ad23928314af28c8e820883c60a55/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.187849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a240ec-345d-4071-82d2-554eff2fe043 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1b894867e354880b1fda81bebdbeae1, user_id=c96ad23928314af28c8e820883c60a55, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.191863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a240ec-345d-4071-82d2-554eff2fe043 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.204411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1020/2045] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:42 2022] PUT /identity/v3/projects/b1b894867e354880b1fda81bebdbeae1/users/c96ad23928314af28c8e820883c60a55/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:42.206886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1026/2046] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:42 2022] PUT /identity/v3/projects/c2da67bdb084426ea652d7ff328c42bb/users/29f5e2c344cc4b17be1b175e934a9245/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:42.214602 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7864ed-8a94-4f45-ab61-f39443ee35ee None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.214889 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7864ed-8a94-4f45-ab61-f39443ee35ee None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.215161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7864ed-8a94-4f45-ab61-f39443ee35ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.219998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9593559-aa2a-4a04-8147-573b7e6f477c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.220574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9593559-aa2a-4a04-8147-573b7e6f477c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.221067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9593559-aa2a-4a04-8147-573b7e6f477c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.246412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-1a7864ed-8a94-4f45-ab61-f39443ee35ee None None] MFA Rules not processed for user `c96ad23928314af28c8e820883c60a55`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:42.247644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1a7864ed-8a94-4f45-ab61-f39443ee35ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:42.248099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:42.267799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d9593559-aa2a-4a04-8147-573b7e6f477c None None] MFA Rules not processed for user `29f5e2c344cc4b17be1b175e934a9245`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:42.269418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d9593559-aa2a-4a04-8147-573b7e6f477c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:42.307833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:42.373575 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1021/2047] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:42 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:42.374352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1027/2048] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:42 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:42.375483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7864ed-8a94-4f45-ab61-f39443ee35ee None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.460067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7864ed-8a94-4f45-ab61-f39443ee35ee None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:42.466953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bd028f-fb1f-4232-970d-f0a91f344a2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:42.467770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bd028f-fb1f-4232-970d-f0a91f344a2b admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.468205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bd028f-fb1f-4232-970d-f0a91f344a2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.468685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bd028f-fb1f-4232-970d-f0a91f344a2b admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.469463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45bd028f-fb1f-4232-970d-f0a91f344a2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.471642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45bd028f-fb1f-4232-970d-f0a91f344a2b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.510931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1022/2049] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:42 2022] POST /identity/v3/projects => generated 385 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:42.514983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9593559-aa2a-4a04-8147-573b7e6f477c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.529114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9593559-aa2a-4a04-8147-573b7e6f477c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:42.574319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bd028f-fb1f-4232-970d-f0a91f344a2b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.632925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba472f7e-1916-4a69-89bc-e3811741dc06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:42.633674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba472f7e-1916-4a69-89bc-e3811741dc06 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.634000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba472f7e-1916-4a69-89bc-e3811741dc06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.634298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba472f7e-1916-4a69-89bc-e3811741dc06 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.634931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba472f7e-1916-4a69-89bc-e3811741dc06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.639422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba472f7e-1916-4a69-89bc-e3811741dc06 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.674771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1028/2050] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:42 2022] POST /identity/v3/users => generated 293 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:42.676206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba472f7e-1916-4a69-89bc-e3811741dc06 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.684947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bd028f-fb1f-4232-970d-f0a91f344a2b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:42.688954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba472f7e-1916-4a69-89bc-e3811741dc06 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:42.689960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3116abed-1802-441b-b0e8-709c90f3d912 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:42.691203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3116abed-1802-441b-b0e8-709c90f3d912 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.691561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3116abed-1802-441b-b0e8-709c90f3d912 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.691900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3116abed-1802-441b-b0e8-709c90f3d912 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.695172 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3da35f-2b46-42aa-874d-0e23eaca00c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:42.695776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3da35f-2b46-42aa-874d-0e23eaca00c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.696049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3da35f-2b46-42aa-874d-0e23eaca00c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.696314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3da35f-2b46-42aa-874d-0e23eaca00c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.697406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3da35f-2b46-42aa-874d-0e23eaca00c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.699528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3da35f-2b46-42aa-874d-0e23eaca00c5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.712050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1029/2051] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:42 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:42.715721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3da35f-2b46-42aa-874d-0e23eaca00c5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.728802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3da35f-2b46-42aa-874d-0e23eaca00c5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:42.734639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a878bbae-144a-442c-ae76-2136d22a8e88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:42.735435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a878bbae-144a-442c-ae76-2136d22a8e88 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.735780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a878bbae-144a-442c-ae76-2136d22a8e88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.736168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a878bbae-144a-442c-ae76-2136d22a8e88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.737964 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a878bbae-144a-442c-ae76-2136d22a8e88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.739945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a878bbae-144a-442c-ae76-2136d22a8e88 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.754544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1030/2052] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:42 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:42.758290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a878bbae-144a-442c-ae76-2136d22a8e88 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.770577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a878bbae-144a-442c-ae76-2136d22a8e88 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:42.776220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3482ee1d-4463-4309-be30-625c3656e649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:42.777081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3482ee1d-4463-4309-be30-625c3656e649 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.777499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3482ee1d-4463-4309-be30-625c3656e649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.777926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3482ee1d-4463-4309-be30-625c3656e649 admin admin] PATH_INFO: `/v3/projects/9d7581e33ce04f3eba505fdf077e45c0/users/68bfec3a83174c63b22fff74bdf45765/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.782630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3116abed-1802-441b-b0e8-709c90f3d912 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.784433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3116abed-1802-441b-b0e8-709c90f3d912 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.795024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3482ee1d-4463-4309-be30-625c3656e649 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d7581e33ce04f3eba505fdf077e45c0, user_id=68bfec3a83174c63b22fff74bdf45765, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:42.795982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1023/2053] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:42 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:42.804932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3482ee1d-4463-4309-be30-625c3656e649 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:42.842439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82fc75e-b7d9-44b8-8131-b6d1a510de67 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:42.842439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82fc75e-b7d9-44b8-8131-b6d1a510de67 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:42.842439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82fc75e-b7d9-44b8-8131-b6d1a510de67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:42.844012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1031/2054] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:42 2022] PUT /identity/v3/projects/9d7581e33ce04f3eba505fdf077e45c0/users/68bfec3a83174c63b22fff74bdf45765/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:42.851847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:42.884017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e82fc75e-b7d9-44b8-8131-b6d1a510de67 None None] MFA Rules not processed for user `68bfec3a83174c63b22fff74bdf45765`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:42.885289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e82fc75e-b7d9-44b8-8131-b6d1a510de67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:42.962486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3482ee1d-4463-4309-be30-625c3656e649 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:42.964340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3482ee1d-4463-4309-be30-625c3656e649 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:42.968237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1024/2055] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:42 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:42.969768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82fc75e-b7d9-44b8-8131-b6d1a510de67 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:43.059176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:43.083851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3482ee1d-4463-4309-be30-625c3656e649 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:43.090979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:43.091814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.092158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.092497 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] PATH_INFO: `/v3/users/ce8b1e6dec944b068173bb8c920efe5a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.100762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce8b1e6dec944b068173bb8c920efe5a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:43.100762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:43.146842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82fc75e-b7d9-44b8-8131-b6d1a510de67 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:43.151141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90757866-c15b-405d-a62f-70fdf0332e52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:43.152153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90757866-c15b-405d-a62f-70fdf0332e52 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.152513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90757866-c15b-405d-a62f-70fdf0332e52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.152997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90757866-c15b-405d-a62f-70fdf0332e52 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.154046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90757866-c15b-405d-a62f-70fdf0332e52 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:43.156212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90757866-c15b-405d-a62f-70fdf0332e52 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:43.199534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce8b1e6dec944b068173bb8c920efe5a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:43.203583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1025/2056] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:42 2022] POST /identity/v3/projects => generated 385 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:43.210571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90757866-c15b-405d-a62f-70fdf0332e52 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:43.211992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce8b1e6dec944b068173bb8c920efe5a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:43.212831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce8b1e6dec944b068173bb8c920efe5a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:43.213492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce8b1e6dec944b068173bb8c920efe5a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:43.224058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ade2a83e-6388-4158-a613-66aeeb8ef1fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce8b1e6dec944b068173bb8c920efe5a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:43.255130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1032/2057] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:42 2022] DELETE /identity/v3/users/ce8b1e6dec944b068173bb8c920efe5a => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:43.255130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598c4351-a14b-4989-9b43-6e59d61fb338 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.255130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598c4351-a14b-4989-9b43-6e59d61fb338 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.255130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598c4351-a14b-4989-9b43-6e59d61fb338 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.285245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-598c4351-a14b-4989-9b43-6e59d61fb338 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:43.286537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-598c4351-a14b-4989-9b43-6e59d61fb338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:43.364829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90757866-c15b-405d-a62f-70fdf0332e52 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:43.402617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1033/2058] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:43.410803 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598c4351-a14b-4989-9b43-6e59d61fb338 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:43.413207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-598c4351-a14b-4989-9b43-6e59d61fb338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:43.430681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:43.472861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a40b09-cd33-4ced-a16b-daf847612842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:43.473770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a40b09-cd33-4ced-a16b-daf847612842 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.474610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a40b09-cd33-4ced-a16b-daf847612842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.474900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a40b09-cd33-4ced-a16b-daf847612842 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.475710 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a40b09-cd33-4ced-a16b-daf847612842 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:43.478695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a40b09-cd33-4ced-a16b-daf847612842 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:43.507053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1026/2059] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:43 2022] POST /identity/v3/users => generated 299 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:43.507981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a40b09-cd33-4ced-a16b-daf847612842 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:43.511527 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-02a40b09-cd33-4ced-a16b-daf847612842 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:43.515419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598c4351-a14b-4989-9b43-6e59d61fb338 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:43.520194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71de288-ec20-4e9f-b325-692b51ecb6a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:43.521044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71de288-ec20-4e9f-b325-692b51ecb6a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.521367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71de288-ec20-4e9f-b325-692b51ecb6a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.521784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71de288-ec20-4e9f-b325-692b51ecb6a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.527017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a40b09-cd33-4ced-a16b-daf847612842 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:43.532408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71de288-ec20-4e9f-b325-692b51ecb6a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:43.533883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:43.534049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71de288-ec20-4e9f-b325-692b51ecb6a0 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:43.534588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.534943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.535414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] PATH_INFO: `/v3/users/0702c01019ee4a358b9243c0e38cdc20` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.536533 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0702c01019ee4a358b9243c0e38cdc20)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:43.538587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:43.545946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1034/2060] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:43.547014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71de288-ec20-4e9f-b325-692b51ecb6a0 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:43.548397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d71de288-ec20-4e9f-b325-692b51ecb6a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:43.565705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:43.573868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71de288-ec20-4e9f-b325-692b51ecb6a0 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:43.590261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d322a2-d754-4f19-bc80-5f4d6ff3f941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:43.590935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d322a2-d754-4f19-bc80-5f4d6ff3f941 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.591949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d322a2-d754-4f19-bc80-5f4d6ff3f941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.592212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d322a2-d754-4f19-bc80-5f4d6ff3f941 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.593548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d322a2-d754-4f19-bc80-5f4d6ff3f941 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:43.604707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d322a2-d754-4f19-bc80-5f4d6ff3f941 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:43.630121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1035/2061] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:43 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:43.633502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d322a2-d754-4f19-bc80-5f4d6ff3f941 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:43.635610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0702c01019ee4a358b9243c0e38cdc20'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:43.636848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0702c01019ee4a358b9243c0e38cdc20'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:43.647875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0702c01019ee4a358b9243c0e38cdc20'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:43.649210 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d322a2-d754-4f19-bc80-5f4d6ff3f941 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:43.649713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0702c01019ee4a358b9243c0e38cdc20'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:43.655246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257a8ee7-de8f-4b21-8bd0-e20b35009e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:43.655936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257a8ee7-de8f-4b21-8bd0-e20b35009e17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.656249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257a8ee7-de8f-4b21-8bd0-e20b35009e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.656476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257a8ee7-de8f-4b21-8bd0-e20b35009e17 admin admin] PATH_INFO: `/v3/projects/9ea6439ee303450a93934a5799aee0fd/users/07655783f09c449b94b1c85ec33fd4f0/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.661481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3911ed3d-f2aa-4a2b-8eb5-3d1fba807cb5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0702c01019ee4a358b9243c0e38cdc20'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:43.679655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257a8ee7-de8f-4b21-8bd0-e20b35009e17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ea6439ee303450a93934a5799aee0fd, user_id=07655783f09c449b94b1c85ec33fd4f0, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:43.683996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257a8ee7-de8f-4b21-8bd0-e20b35009e17 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:43.709165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1027/2062] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:43 2022] DELETE /identity/v3/users/0702c01019ee4a358b9243c0e38cdc20 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:43.715529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06919e24-2801-4561-b6b9-eb5cb7eceaad None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.717485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1036/2063] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:43 2022] PUT /identity/v3/projects/9ea6439ee303450a93934a5799aee0fd/users/07655783f09c449b94b1c85ec33fd4f0/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:43.717972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06919e24-2801-4561-b6b9-eb5cb7eceaad None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.718439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06919e24-2801-4561-b6b9-eb5cb7eceaad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.721141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257a8ee7-de8f-4b21-8bd0-e20b35009e17 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:43.723193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-257a8ee7-de8f-4b21-8bd0-e20b35009e17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:43.744946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-06919e24-2801-4561-b6b9-eb5cb7eceaad None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:43.746565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-06919e24-2801-4561-b6b9-eb5cb7eceaad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:43.829104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257a8ee7-de8f-4b21-8bd0-e20b35009e17 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:43.892443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1028/2064] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:43.910242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06919e24-2801-4561-b6b9-eb5cb7eceaad None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:43.912808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-06919e24-2801-4561-b6b9-eb5cb7eceaad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:43.931876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3592e6fb-1d5c-49a8-9c21-6c2d37d1a6d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:43.933033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3592e6fb-1d5c-49a8-9c21-6c2d37d1a6d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.933033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3592e6fb-1d5c-49a8-9c21-6c2d37d1a6d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.933532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3592e6fb-1d5c-49a8-9c21-6c2d37d1a6d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.935416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3592e6fb-1d5c-49a8-9c21-6c2d37d1a6d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:43.937412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3592e6fb-1d5c-49a8-9c21-6c2d37d1a6d9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:43.951933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1037/2065] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:43 2022] GET /identity/v3/roles => generated 1420 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:43.955927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3592e6fb-1d5c-49a8-9c21-6c2d37d1a6d9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:43.970979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3592e6fb-1d5c-49a8-9c21-6c2d37d1a6d9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:43.977998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3841278-a5e7-4364-b932-4ec6f61f12c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:43.979018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3841278-a5e7-4364-b932-4ec6f61f12c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:43.979932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3841278-a5e7-4364-b932-4ec6f61f12c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:43.980329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3841278-a5e7-4364-b932-4ec6f61f12c4 admin admin] PATH_INFO: `/v3/projects/9ea6439ee303450a93934a5799aee0fd/users/07655783f09c449b94b1c85ec33fd4f0/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:43.984278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3841278-a5e7-4364-b932-4ec6f61f12c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ea6439ee303450a93934a5799aee0fd, user_id=07655783f09c449b94b1c85ec33fd4f0, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:43.994683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3841278-a5e7-4364-b932-4ec6f61f12c4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:43.995684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:44.028342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1038/2066] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:43 2022] PUT /identity/v3/projects/9ea6439ee303450a93934a5799aee0fd/users/07655783f09c449b94b1c85ec33fd4f0/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:44.061312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4d01f-6724-43b1-8398-027a3a59ebcd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:44.061312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4d01f-6724-43b1-8398-027a3a59ebcd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:44.061312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4d01f-6724-43b1-8398-027a3a59ebcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:44.063546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06919e24-2801-4561-b6b9-eb5cb7eceaad None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:44.136080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-59b4d01f-6724-43b1-8398-027a3a59ebcd None None] MFA Rules not processed for user `07655783f09c449b94b1c85ec33fd4f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:44.137500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-59b4d01f-6724-43b1-8398-027a3a59ebcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:44.194000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e196171-6f3f-4557-b7c3-8887f2c6351d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:44.195763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e196171-6f3f-4557-b7c3-8887f2c6351d service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:44.196211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e196171-6f3f-4557-b7c3-8887f2c6351d service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:44.196508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e196171-6f3f-4557-b7c3-8887f2c6351d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:44.218679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:44.249472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1039/2067] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:44.263374 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4d01f-6724-43b1-8398-027a3a59ebcd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:44.264775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-59b4d01f-6724-43b1-8398-027a3a59ebcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:44.309727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e196171-6f3f-4557-b7c3-8887f2c6351d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:44.311456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e196171-6f3f-4557-b7c3-8887f2c6351d service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:44.324796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1029/2068] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:44.345056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4d01f-6724-43b1-8398-027a3a59ebcd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:44.355293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f092ee-3a30-4926-812f-300245ed82d7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:44.357240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f092ee-3a30-4926-812f-300245ed82d7 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:44.357881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f092ee-3a30-4926-812f-300245ed82d7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:44.358859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f092ee-3a30-4926-812f-300245ed82d7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:44.362651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e1f092ee-3a30-4926-812f-300245ed82d7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:44.409181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e196171-6f3f-4557-b7c3-8887f2c6351d service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:44.412491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0e196171-6f3f-4557-b7c3-8887f2c6351d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:44.437100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e196171-6f3f-4557-b7c3-8887f2c6351d service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:44.443594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:44.444542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:44.445112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:44.445251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] PATH_INFO: `/v3/projects/548745ac31cf449bbbad9dff9e3cd4f1` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:44.449083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=548745ac31cf449bbbad9dff9e3cd4f1)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:44.451819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:44.491238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f092ee-3a30-4926-812f-300245ed82d7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:44.493750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f092ee-3a30-4926-812f-300245ed82d7 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:44.510746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1040/2069] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:44 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:44.539384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '548745ac31cf449bbbad9dff9e3cd4f1', 'request_id': 'req-ab2d1eff-9898-4819-a432-aacbcaf275f6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.550588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '548745ac31cf449bbbad9dff9e3cd4f1', 'request_id': 'req-ab2d1eff-9898-4819-a432-aacbcaf275f6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.562263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '548745ac31cf449bbbad9dff9e3cd4f1', 'request_id': 'req-ab2d1eff-9898-4819-a432-aacbcaf275f6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.582043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] The token cache is being invalidate because project 548745ac31cf449bbbad9dff9e3cd4f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:44.582694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.585055 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.591651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f092ee-3a30-4926-812f-300245ed82d7 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:44.593004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1030/2070] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:44 2022] DELETE /identity/v3/projects/548745ac31cf449bbbad9dff9e3cd4f1 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:44.606220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f092ee-3a30-4926-812f-300245ed82d7 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:44.720588 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:44.721306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:44.722833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:44.723200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] PATH_INFO: `/v3/users/52086fdaf1334c42a5a3503a0bcf64c7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:44.727384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] RBAC: Authorizing `identity:delete_user(user_id=52086fdaf1334c42a5a3503a0bcf64c7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:44.734040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:44.822650 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:44.828905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52086fdaf1334c42a5a3503a0bcf64c7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.842293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52086fdaf1334c42a5a3503a0bcf64c7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.843381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52086fdaf1334c42a5a3503a0bcf64c7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.844411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52086fdaf1334c42a5a3503a0bcf64c7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.850355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:44.851840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:44.861962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52086fdaf1334c42a5a3503a0bcf64c7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.879013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d1eff-9898-4819-a432-aacbcaf275f6 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:44.887911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:44.889001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:44.889613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:44.890374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] PATH_INFO: `/v3/projects/ce60d6a8cea344a3b7d2e332fcfeb8fc` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:44.891574 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce60d6a8cea344a3b7d2e332fcfeb8fc)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:44.896212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:44.897060 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1041/2071] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:44 2022] DELETE /identity/v3/users/52086fdaf1334c42a5a3503a0bcf64c7 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:44.926032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:44.985541 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce60d6a8cea344a3b7d2e332fcfeb8fc', 'request_id': 'req-a2603f6a-0889-460b-a93b-9f5abfb05d50'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:44.997197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce60d6a8cea344a3b7d2e332fcfeb8fc', 'request_id': 'req-a2603f6a-0889-460b-a93b-9f5abfb05d50'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.007395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce60d6a8cea344a3b7d2e332fcfeb8fc', 'request_id': 'req-a2603f6a-0889-460b-a93b-9f5abfb05d50'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.023883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] The token cache is being invalidate because project ce60d6a8cea344a3b7d2e332fcfeb8fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:45.024313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.025922 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.028137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1031/2072] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:44 2022] DELETE /identity/v3/projects/ce60d6a8cea344a3b7d2e332fcfeb8fc => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:45.032392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:45.035240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:45.115934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5a17f4-6963-4c29-97c2-67a4b63b4a3c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:45.207194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:45.207974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:45.208347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:45.208731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] PATH_INFO: `/v3/users/c96ad23928314af28c8e820883c60a55` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:45.209946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c96ad23928314af28c8e820883c60a55)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:45.212875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:45.272861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c96ad23928314af28c8e820883c60a55'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.281156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c96ad23928314af28c8e820883c60a55'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.281948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c96ad23928314af28c8e820883c60a55'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.282752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c96ad23928314af28c8e820883c60a55'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.294658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c509e250-4d44-448f-8dd5-30db00b52728 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c96ad23928314af28c8e820883c60a55'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.300950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:45.319478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1042/2073] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:45 2022] DELETE /identity/v3/users/c96ad23928314af28c8e820883c60a55 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:45.411392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:45.412863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:45.426723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2603f6a-0889-460b-a93b-9f5abfb05d50 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:45.508116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:45.508837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:45.509215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:45.509560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] PATH_INFO: `/v3/projects/b8258ea4cfb9402dbac50d953fd2cbf8` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:45.511276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8258ea4cfb9402dbac50d953fd2cbf8)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:45.513319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:45.577864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8258ea4cfb9402dbac50d953fd2cbf8', 'request_id': 'req-f697a3b4-c56d-419c-b0bb-c9947607a0f3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.599158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8258ea4cfb9402dbac50d953fd2cbf8', 'request_id': 'req-f697a3b4-c56d-419c-b0bb-c9947607a0f3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.614127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8258ea4cfb9402dbac50d953fd2cbf8', 'request_id': 'req-f697a3b4-c56d-419c-b0bb-c9947607a0f3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.626584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] The token cache is being invalidate because project b8258ea4cfb9402dbac50d953fd2cbf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:45.626873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.627538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.630207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1032/2074] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:45 2022] DELETE /identity/v3/projects/b8258ea4cfb9402dbac50d953fd2cbf8 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:45.634689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1edf92a-e7cb-4ec4-b04e-9f7e51cb479e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:45.635156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1edf92a-e7cb-4ec4-b04e-9f7e51cb479e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:45.635236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1edf92a-e7cb-4ec4-b04e-9f7e51cb479e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:45.655632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f1edf92a-e7cb-4ec4-b04e-9f7e51cb479e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:45.656775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f1edf92a-e7cb-4ec4-b04e-9f7e51cb479e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:45.700690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:45.702508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:45.728312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1043/2075] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:45.739968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6d2810-30ed-47a7-bafe-6f1b817876c9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:45.740329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6d2810-30ed-47a7-bafe-6f1b817876c9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:45.740612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6d2810-30ed-47a7-bafe-6f1b817876c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:45.752758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:45.763968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5b6d2810-30ed-47a7-bafe-6f1b817876c9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:45.765212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5b6d2810-30ed-47a7-bafe-6f1b817876c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:45.774483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1044/2076] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:45.778535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6d2810-30ed-47a7-bafe-6f1b817876c9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:45.788046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6d2810-30ed-47a7-bafe-6f1b817876c9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:45.790910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f697a3b4-c56d-419c-b0bb-c9947607a0f3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:45.794813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd4968a-2083-487c-a46f-a10587ea65f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:45.795091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd4968a-2083-487c-a46f-a10587ea65f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:45.795368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd4968a-2083-487c-a46f-a10587ea65f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:45.795670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd4968a-2083-487c-a46f-a10587ea65f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:45.796466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd4968a-2083-487c-a46f-a10587ea65f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:45.796774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd4968a-2083-487c-a46f-a10587ea65f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:45.798568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd4968a-2083-487c-a46f-a10587ea65f9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:45.811328 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:45.812108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:45.812480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:45.812853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] PATH_INFO: `/v3/projects/b1b894867e354880b1fda81bebdbeae1` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:45.814013 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1045/2077] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:55:45.817667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1b894867e354880b1fda81bebdbeae1)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:45.818272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd4968a-2083-487c-a46f-a10587ea65f9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:45.820079 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:45.828068 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd4968a-2083-487c-a46f-a10587ea65f9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:45.833408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a40fbb-eb45-4b98-924f-1b2eb1505269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:45.834104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a40fbb-eb45-4b98-924f-1b2eb1505269 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:45.834495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a40fbb-eb45-4b98-924f-1b2eb1505269 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:45.834842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a40fbb-eb45-4b98-924f-1b2eb1505269 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:45.835667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a40fbb-eb45-4b98-924f-1b2eb1505269 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:45.837762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a40fbb-eb45-4b98-924f-1b2eb1505269 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:45.885824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1046/2078] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:45 2022] POST /identity/v3/projects => generated 385 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:45.889961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a40fbb-eb45-4b98-924f-1b2eb1505269 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:45.902913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1b894867e354880b1fda81bebdbeae1', 'request_id': 'req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.912386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1b894867e354880b1fda81bebdbeae1', 'request_id': 'req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.920288 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1b894867e354880b1fda81bebdbeae1', 'request_id': 'req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.934823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] The token cache is being invalidate because project b1b894867e354880b1fda81bebdbeae1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:45.935122 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.935810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e3ce1e-933f-40e4-abe0-5a9c8c00bffb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:45.940910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1033/2079] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:45 2022] DELETE /identity/v3/projects/b1b894867e354880b1fda81bebdbeae1 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:45.942443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67b3953-c2e1-4c6a-85b4-a796c4af1507 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:45.942726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67b3953-c2e1-4c6a-85b4-a796c4af1507 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:45.943000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67b3953-c2e1-4c6a-85b4-a796c4af1507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:45.966172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-d67b3953-c2e1-4c6a-85b4-a796c4af1507 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:45.967407 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d67b3953-c2e1-4c6a-85b4-a796c4af1507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:45.967539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a40fbb-eb45-4b98-924f-1b2eb1505269 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:45.972518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4166c1-8c17-45bc-ab45-4bb0c6878cdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:45.973127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4166c1-8c17-45bc-ab45-4bb0c6878cdd admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:45.973405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4166c1-8c17-45bc-ab45-4bb0c6878cdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:45.973678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4166c1-8c17-45bc-ab45-4bb0c6878cdd admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:45.974422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4166c1-8c17-45bc-ab45-4bb0c6878cdd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:45.976213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1034/2080] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:45.976353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4166c1-8c17-45bc-ab45-4bb0c6878cdd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:45.983668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af985152-f065-4199-a84c-c0cb422112cb None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:45.983949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af985152-f065-4199-a84c-c0cb422112cb None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:45.984219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af985152-f065-4199-a84c-c0cb422112cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.009797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1047/2081] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:45 2022] POST /identity/v3/users => generated 293 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:46.011888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-af985152-f065-4199-a84c-c0cb422112cb None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:46.014028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-af985152-f065-4199-a84c-c0cb422112cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:46.014362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4166c1-8c17-45bc-ab45-4bb0c6878cdd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.016809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0d4166c1-8c17-45bc-ab45-4bb0c6878cdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:46.031268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1035/2082] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:46.031268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af985152-f065-4199-a84c-c0cb422112cb None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.032549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4166c1-8c17-45bc-ab45-4bb0c6878cdd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.041306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af985152-f065-4199-a84c-c0cb422112cb None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.042999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1a7523-0abd-4002-87a3-ab0263a93aef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.043489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1a7523-0abd-4002-87a3-ab0263a93aef admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.043863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1a7523-0abd-4002-87a3-ab0263a93aef admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.044343 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1a7523-0abd-4002-87a3-ab0263a93aef admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.045201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1a7523-0abd-4002-87a3-ab0263a93aef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.047123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7232ce39-4abf-4c87-9a7f-9e66c0a38b45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.047360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1a7523-0abd-4002-87a3-ab0263a93aef admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.047899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7232ce39-4abf-4c87-9a7f-9e66c0a38b45 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.049054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7232ce39-4abf-4c87-9a7f-9e66c0a38b45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.049405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7232ce39-4abf-4c87-9a7f-9e66c0a38b45 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.050536 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7232ce39-4abf-4c87-9a7f-9e66c0a38b45 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:46.050847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7232ce39-4abf-4c87-9a7f-9e66c0a38b45 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.052632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7232ce39-4abf-4c87-9a7f-9e66c0a38b45 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.069782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1048/2083] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:46 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:46.070997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1a7523-0abd-4002-87a3-ab0263a93aef admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.079660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1036/2084] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:46.080731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7232ce39-4abf-4c87-9a7f-9e66c0a38b45 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.081514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1a7523-0abd-4002-87a3-ab0263a93aef admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.086100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e5814c-b562-4a7c-baab-4ec29f6c00ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.086944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e5814c-b562-4a7c-baab-4ec29f6c00ea admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.087076 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e5814c-b562-4a7c-baab-4ec29f6c00ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.087248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e5814c-b562-4a7c-baab-4ec29f6c00ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.088006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e5814c-b562-4a7c-baab-4ec29f6c00ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.089837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e5814c-b562-4a7c-baab-4ec29f6c00ea admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.092204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7232ce39-4abf-4c87-9a7f-9e66c0a38b45 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.099614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8067dc4-96f6-4746-bf0c-39948207cd69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.100224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8067dc4-96f6-4746-bf0c-39948207cd69 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.100531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8067dc4-96f6-4746-bf0c-39948207cd69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.100839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8067dc4-96f6-4746-bf0c-39948207cd69 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.101191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1049/2085] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:46.102742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8067dc4-96f6-4746-bf0c-39948207cd69 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.104699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8067dc4-96f6-4746-bf0c-39948207cd69 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.105139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e5814c-b562-4a7c-baab-4ec29f6c00ea admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.117276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e5814c-b562-4a7c-baab-4ec29f6c00ea admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.123190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8137a1-c139-490d-80d5-118e1f414562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.123970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8137a1-c139-490d-80d5-118e1f414562 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.124382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8137a1-c139-490d-80d5-118e1f414562 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.124744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8137a1-c139-490d-80d5-118e1f414562 admin admin] PATH_INFO: `/v3/projects/0e5ff02c2cb6476abf55812f209afc9f/users/3e7226a987184c1d9dcdc248d0f4991b/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.134040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:46.138460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1037/2086] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:46 2022] POST /identity/v3/projects => generated 363 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:46.139957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8067dc4-96f6-4746-bf0c-39948207cd69 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.143808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8137a1-c139-490d-80d5-118e1f414562 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e5ff02c2cb6476abf55812f209afc9f, user_id=3e7226a987184c1d9dcdc248d0f4991b, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.146998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8137a1-c139-490d-80d5-118e1f414562 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.167555 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1050/2087] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:46 2022] PUT /identity/v3/projects/0e5ff02c2cb6476abf55812f209afc9f/users/3e7226a987184c1d9dcdc248d0f4991b/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:46.179416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05de18ba-5b10-4183-9935-3dc8e2d8a75c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.179882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05de18ba-5b10-4183-9935-3dc8e2d8a75c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.179974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05de18ba-5b10-4183-9935-3dc8e2d8a75c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.215158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-05de18ba-5b10-4183-9935-3dc8e2d8a75c None None] MFA Rules not processed for user `3e7226a987184c1d9dcdc248d0f4991b`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:46.216365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-05de18ba-5b10-4183-9935-3dc8e2d8a75c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:46.227739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8067dc4-96f6-4746-bf0c-39948207cd69 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.245990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:46.281615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1051/2088] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:46.287947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283644d1-36cf-44b2-8c84-35e04823d500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.288569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283644d1-36cf-44b2-8c84-35e04823d500 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.288848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283644d1-36cf-44b2-8c84-35e04823d500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.289118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283644d1-36cf-44b2-8c84-35e04823d500 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.289742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283644d1-36cf-44b2-8c84-35e04823d500 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.291706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283644d1-36cf-44b2-8c84-35e04823d500 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.292879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05de18ba-5b10-4183-9935-3dc8e2d8a75c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.294280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-05de18ba-5b10-4183-9935-3dc8e2d8a75c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:46.314015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1038/2089] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:46 2022] POST /identity/v3/users => generated 282 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:46.318372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283644d1-36cf-44b2-8c84-35e04823d500 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.327662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283644d1-36cf-44b2-8c84-35e04823d500 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.332148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b2a97d-03fd-46ea-960f-08801234f456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.332727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b2a97d-03fd-46ea-960f-08801234f456 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.333003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b2a97d-03fd-46ea-960f-08801234f456 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.333272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b2a97d-03fd-46ea-960f-08801234f456 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.333896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b2a97d-03fd-46ea-960f-08801234f456 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.335548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b2a97d-03fd-46ea-960f-08801234f456 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.345290 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1039/2090] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:46 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:46.349711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b2a97d-03fd-46ea-960f-08801234f456 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.352224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05de18ba-5b10-4183-9935-3dc8e2d8a75c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.355905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e44d8c-a44d-4fb5-8661-f21c0956f71d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.356506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e44d8c-a44d-4fb5-8661-f21c0956f71d service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.356783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e44d8c-a44d-4fb5-8661-f21c0956f71d service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.357059 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e44d8c-a44d-4fb5-8661-f21c0956f71d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.359476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b2a97d-03fd-46ea-960f-08801234f456 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.364104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03628d2-83bf-4090-aba9-45e812fbac93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.364680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03628d2-83bf-4090-aba9-45e812fbac93 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.364969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03628d2-83bf-4090-aba9-45e812fbac93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.365240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03628d2-83bf-4090-aba9-45e812fbac93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.365617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e44d8c-a44d-4fb5-8661-f21c0956f71d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.365860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03628d2-83bf-4090-aba9-45e812fbac93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.367070 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e44d8c-a44d-4fb5-8661-f21c0956f71d service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.367452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03628d2-83bf-4090-aba9-45e812fbac93 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.376566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1052/2091] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:46 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:46.378865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1040/2092] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:46.382855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e44d8c-a44d-4fb5-8661-f21c0956f71d service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.393063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e44d8c-a44d-4fb5-8661-f21c0956f71d service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.397685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad4235f-4747-4e3e-8dd2-c24ff9354063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.398362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad4235f-4747-4e3e-8dd2-c24ff9354063 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.398637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad4235f-4747-4e3e-8dd2-c24ff9354063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.398907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad4235f-4747-4e3e-8dd2-c24ff9354063 admin admin] PATH_INFO: `/v3/projects/30703798604e4e8f9ba00a8df2fd4dd6/users/21d93fecf33c4aecbd3ccf1f9f558fcb/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.417066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad4235f-4747-4e3e-8dd2-c24ff9354063 admin admin] RBAC: Authorizing `identity:create_grant(project_id=30703798604e4e8f9ba00a8df2fd4dd6, user_id=21d93fecf33c4aecbd3ccf1f9f558fcb, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.420881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad4235f-4747-4e3e-8dd2-c24ff9354063 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.435521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1053/2093] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:46 2022] PUT /identity/v3/projects/30703798604e4e8f9ba00a8df2fd4dd6/users/21d93fecf33c4aecbd3ccf1f9f558fcb/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:46.447795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa616bf7-617e-48dc-b234-9c1013e215bf None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.448109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa616bf7-617e-48dc-b234-9c1013e215bf None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.448528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa616bf7-617e-48dc-b234-9c1013e215bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.477143 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-aa616bf7-617e-48dc-b234-9c1013e215bf None None] MFA Rules not processed for user `21d93fecf33c4aecbd3ccf1f9f558fcb`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:46.478508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-aa616bf7-617e-48dc-b234-9c1013e215bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:46.559534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1041/2094] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:46.564246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad4235f-4747-4e3e-8dd2-c24ff9354063 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.634969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:46.649884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad4235f-4747-4e3e-8dd2-c24ff9354063 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.654561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1eba511-8b46-45ef-8ef2-a67015ceb2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.655333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eba511-8b46-45ef-8ef2-a67015ceb2cf admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.655677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eba511-8b46-45ef-8ef2-a67015ceb2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.656005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1eba511-8b46-45ef-8ef2-a67015ceb2cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.656690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1eba511-8b46-45ef-8ef2-a67015ceb2cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.658686 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1eba511-8b46-45ef-8ef2-a67015ceb2cf admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.687015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1054/2095] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:46 2022] POST /identity/v3/projects => generated 365 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:46.692372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa616bf7-617e-48dc-b234-9c1013e215bf None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.706333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa616bf7-617e-48dc-b234-9c1013e215bf None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.767577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5696d7-876c-47de-baab-96852c743112 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.768190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5696d7-876c-47de-baab-96852c743112 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.768493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5696d7-876c-47de-baab-96852c743112 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.768768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5696d7-876c-47de-baab-96852c743112 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.769390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5696d7-876c-47de-baab-96852c743112 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.771325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5696d7-876c-47de-baab-96852c743112 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.791648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1042/2096] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:46 2022] POST /identity/v3/users => generated 289 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:46.795571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1eba511-8b46-45ef-8ef2-a67015ceb2cf admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.806449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1eba511-8b46-45ef-8ef2-a67015ceb2cf admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.811163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb6078f-a1cf-4fc5-9ee5-bf12b2dcc44e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.811749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb6078f-a1cf-4fc5-9ee5-bf12b2dcc44e admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.812022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb6078f-a1cf-4fc5-9ee5-bf12b2dcc44e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.812294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb6078f-a1cf-4fc5-9ee5-bf12b2dcc44e admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.812923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb6078f-a1cf-4fc5-9ee5-bf12b2dcc44e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.814541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb6078f-a1cf-4fc5-9ee5-bf12b2dcc44e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.825602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1055/2097] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:46.829648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5696d7-876c-47de-baab-96852c743112 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.839703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5696d7-876c-47de-baab-96852c743112 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.844264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82801ef4-6d9f-427f-98c1-82bb5bba44ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.844907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82801ef4-6d9f-427f-98c1-82bb5bba44ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.845180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82801ef4-6d9f-427f-98c1-82bb5bba44ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.845448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82801ef4-6d9f-427f-98c1-82bb5bba44ad admin admin] PATH_INFO: `/v3/projects/26ecde8251e04541af230eafe62c460b/users/38b94c23f90348a190b2ddedf6366ea3/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.847617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:46.861522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82801ef4-6d9f-427f-98c1-82bb5bba44ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=26ecde8251e04541af230eafe62c460b, user_id=38b94c23f90348a190b2ddedf6366ea3, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.864670 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82801ef4-6d9f-427f-98c1-82bb5bba44ad admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.876777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1043/2098] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:46 2022] PUT /identity/v3/projects/26ecde8251e04541af230eafe62c460b/users/38b94c23f90348a190b2ddedf6366ea3/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:46.880825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb6078f-a1cf-4fc5-9ee5-bf12b2dcc44e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.891043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb6078f-a1cf-4fc5-9ee5-bf12b2dcc44e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.949024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca29d32-e842-4c5b-b660-521a8b1503de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.949622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca29d32-e842-4c5b-b660-521a8b1503de admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.949898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca29d32-e842-4c5b-b660-521a8b1503de admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.950196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca29d32-e842-4c5b-b660-521a8b1503de admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.950825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca29d32-e842-4c5b-b660-521a8b1503de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.952404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca29d32-e842-4c5b-b660-521a8b1503de admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:46.961819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1056/2099] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:46 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:46.965897 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82801ef4-6d9f-427f-98c1-82bb5bba44ad admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:46.976674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82801ef4-6d9f-427f-98c1-82bb5bba44ad admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:46.981293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c82a47a-708c-4a59-bf1d-ed4088b7f74c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:46.982216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c82a47a-708c-4a59-bf1d-ed4088b7f74c admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:46.984437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c82a47a-708c-4a59-bf1d-ed4088b7f74c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:46.984735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c82a47a-708c-4a59-bf1d-ed4088b7f74c admin admin] PATH_INFO: `/v3/projects/26ecde8251e04541af230eafe62c460b/users/38b94c23f90348a190b2ddedf6366ea3/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:46.986712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c82a47a-708c-4a59-bf1d-ed4088b7f74c admin admin] RBAC: Authorizing `identity:create_grant(project_id=26ecde8251e04541af230eafe62c460b, user_id=38b94c23f90348a190b2ddedf6366ea3, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:46.990016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c82a47a-708c-4a59-bf1d-ed4088b7f74c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:47.000749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1044/2100] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:46 2022] PUT /identity/v3/projects/26ecde8251e04541af230eafe62c460b/users/38b94c23f90348a190b2ddedf6366ea3/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:47.011560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86503d38-f247-410a-b3e7-262eebe4c419 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:47.011843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86503d38-f247-410a-b3e7-262eebe4c419 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:47.012114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86503d38-f247-410a-b3e7-262eebe4c419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:47.042561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-86503d38-f247-410a-b3e7-262eebe4c419 None None] MFA Rules not processed for user `38b94c23f90348a190b2ddedf6366ea3`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:47.043763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-86503d38-f247-410a-b3e7-262eebe4c419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:47.111665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1057/2101] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:47 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:47.124043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c82a47a-708c-4a59-bf1d-ed4088b7f74c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:47.177734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c82a47a-708c-4a59-bf1d-ed4088b7f74c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:47.181409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c4ef9-68d0-4716-873e-85959eb606ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:47.182001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c4ef9-68d0-4716-873e-85959eb606ae service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:47.182310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c4ef9-68d0-4716-873e-85959eb606ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:47.182605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c4ef9-68d0-4716-873e-85959eb606ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:47.192240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02c4ef9-68d0-4716-873e-85959eb606ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:47.194018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02c4ef9-68d0-4716-873e-85959eb606ae service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:47.210233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1045/2102] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:47 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:47.284979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86503d38-f247-410a-b3e7-262eebe4c419 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:47.295959 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86503d38-f247-410a-b3e7-262eebe4c419 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:47.300159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1020f2ad-29e9-4b2b-98ff-ab3d419be38a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:47.300777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1020f2ad-29e9-4b2b-98ff-ab3d419be38a service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:47.301190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1020f2ad-29e9-4b2b-98ff-ab3d419be38a service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:47.301359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1020f2ad-29e9-4b2b-98ff-ab3d419be38a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:47.352421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:47.363568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1020f2ad-29e9-4b2b-98ff-ab3d419be38a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:47.365503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1020f2ad-29e9-4b2b-98ff-ab3d419be38a service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:47.375359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1058/2103] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:47 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:48.330133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c4ef9-68d0-4716-873e-85959eb606ae service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:48.339013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c4ef9-68d0-4716-873e-85959eb606ae service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:48.342943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77315032-6685-443f-839b-674a878062b7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:48.343655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77315032-6685-443f-839b-674a878062b7 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:48.343787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77315032-6685-443f-839b-674a878062b7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:48.344050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77315032-6685-443f-839b-674a878062b7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:48.345744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-77315032-6685-443f-839b-674a878062b7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:48.407662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77315032-6685-443f-839b-674a878062b7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:48.409619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77315032-6685-443f-839b-674a878062b7 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:48.418836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1046/2104] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:48 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:49.170528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1020f2ad-29e9-4b2b-98ff-ab3d419be38a service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:49.172852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1020f2ad-29e9-4b2b-98ff-ab3d419be38a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:49.189755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1020f2ad-29e9-4b2b-98ff-ab3d419be38a service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:49.195083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fa0c94-abd3-4dd8-80ef-bac31ead53f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:49.196543 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fa0c94-abd3-4dd8-80ef-bac31ead53f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:49.197067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fa0c94-abd3-4dd8-80ef-bac31ead53f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:49.197634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fa0c94-abd3-4dd8-80ef-bac31ead53f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:49.204368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-06fa0c94-abd3-4dd8-80ef-bac31ead53f0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:49.276183 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fa0c94-abd3-4dd8-80ef-bac31ead53f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:49.277665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06fa0c94-abd3-4dd8-80ef-bac31ead53f0 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:49.288816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1059/2105] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:49 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:49.994091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77315032-6685-443f-839b-674a878062b7 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:49.995497 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-77315032-6685-443f-839b-674a878062b7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:50.060792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77315032-6685-443f-839b-674a878062b7 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:50.065815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:50.066469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:50.066751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:50.067026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] PATH_INFO: `/v3/users/b6672b9ee1e142ee880c06bcf6fa9058` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:50.068139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6672b9ee1e142ee880c06bcf6fa9058)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:50.070043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:50.082297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:50.141003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6672b9ee1e142ee880c06bcf6fa9058'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.141693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6672b9ee1e142ee880c06bcf6fa9058'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.148902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6672b9ee1e142ee880c06bcf6fa9058'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.149543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6672b9ee1e142ee880c06bcf6fa9058'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.157189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6672b9ee1e142ee880c06bcf6fa9058'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.176016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1047/2106] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:49 2022] DELETE /identity/v3/users/b6672b9ee1e142ee880c06bcf6fa9058 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:50.180212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0a8a67-3f0e-4840-a47e-1764223e32be None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:50.180679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0a8a67-3f0e-4840-a47e-1764223e32be None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:50.180771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0a8a67-3f0e-4840-a47e-1764223e32be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:50.199843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ae0a8a67-3f0e-4840-a47e-1764223e32be None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:50.201054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ae0a8a67-3f0e-4840-a47e-1764223e32be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:50.269482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1060/2107] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:50.279000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:50.280381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:50.335911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5153c1a-ba5d-44a6-8ff4-6ba8052da4a4 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:50.339419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1abf17c-10e6-4188-82df-a4a7333c694d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:50.340010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1abf17c-10e6-4188-82df-a4a7333c694d service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:50.340286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1abf17c-10e6-4188-82df-a4a7333c694d service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:50.340553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1abf17c-10e6-4188-82df-a4a7333c694d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:50.349268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1abf17c-10e6-4188-82df-a4a7333c694d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:50.350704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1abf17c-10e6-4188-82df-a4a7333c694d service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:50.359785 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1048/2108] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:50.514365 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0a8a67-3f0e-4840-a47e-1764223e32be None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:50.515562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ae0a8a67-3f0e-4840-a47e-1764223e32be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:50.528232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0a8a67-3f0e-4840-a47e-1764223e32be None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:50.532773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:50.533394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:50.533699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:50.534550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] PATH_INFO: `/v3/users/21d93fecf33c4aecbd3ccf1f9f558fcb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:50.535672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] RBAC: Authorizing `identity:delete_user(user_id=21d93fecf33c4aecbd3ccf1f9f558fcb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:50.537580 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:50.584573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21d93fecf33c4aecbd3ccf1f9f558fcb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.591840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21d93fecf33c4aecbd3ccf1f9f558fcb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.592450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21d93fecf33c4aecbd3ccf1f9f558fcb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.593051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21d93fecf33c4aecbd3ccf1f9f558fcb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.602054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21d93fecf33c4aecbd3ccf1f9f558fcb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.613444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:50.621639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1061/2109] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:50 2022] DELETE /identity/v3/users/21d93fecf33c4aecbd3ccf1f9f558fcb => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:50.626001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f22402b-9552-4f57-9417-12d0099bf265 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:50.626323 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f22402b-9552-4f57-9417-12d0099bf265 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:50.626665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f22402b-9552-4f57-9417-12d0099bf265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:50.643807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:50.645304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:50.649047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-1f22402b-9552-4f57-9417-12d0099bf265 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:50.650395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1f22402b-9552-4f57-9417-12d0099bf265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:50.715627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e19c14-f496-48b8-b064-1a961c66308c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:50.720323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:50.720523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1049/2110] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:50.720960 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:50.721235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:50.721505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] PATH_INFO: `/v3/projects/69f68a58495d4c99820e8f37497964b9` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:50.722690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69f68a58495d4c99820e8f37497964b9)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:50.724757 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:50.730119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f22402b-9552-4f57-9417-12d0099bf265 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:50.731697 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1f22402b-9552-4f57-9417-12d0099bf265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:50.770646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:50.773575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69f68a58495d4c99820e8f37497964b9', 'request_id': 'req-190c81d5-c270-4a40-a3cb-baa6940b5498'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.781540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69f68a58495d4c99820e8f37497964b9', 'request_id': 'req-190c81d5-c270-4a40-a3cb-baa6940b5498'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.788672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69f68a58495d4c99820e8f37497964b9', 'request_id': 'req-190c81d5-c270-4a40-a3cb-baa6940b5498'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.794541 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f22402b-9552-4f57-9417-12d0099bf265 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:50.799610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] The token cache is being invalidate because project 69f68a58495d4c99820e8f37497964b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:50.799940 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.800717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.802280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1062/2111] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:50 2022] DELETE /identity/v3/projects/69f68a58495d4c99820e8f37497964b9 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:50.806150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:50.807477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:50.848339 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64728df3-d687-4b03-acba-c9452134a106 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:50.849190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64728df3-d687-4b03-acba-c9452134a106 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:50.849473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64728df3-d687-4b03-acba-c9452134a106 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:50.849744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64728df3-d687-4b03-acba-c9452134a106 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:50.851542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-64728df3-d687-4b03-acba-c9452134a106 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:50.872270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190c81d5-c270-4a40-a3cb-baa6940b5498 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:50.876925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:50.878148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:50.878431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:50.878703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] PATH_INFO: `/v3/users/be87a99b1ede422aa402d373cb289702` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:50.879906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be87a99b1ede422aa402d373cb289702)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:50.881964 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:50.914679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:50.919261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64728df3-d687-4b03-acba-c9452134a106 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:50.920923 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64728df3-d687-4b03-acba-c9452134a106 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:50.930966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1050/2112] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:50.937925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be87a99b1ede422aa402d373cb289702'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.945609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be87a99b1ede422aa402d373cb289702'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.946334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be87a99b1ede422aa402d373cb289702'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.946956 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be87a99b1ede422aa402d373cb289702'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.955359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be87a99b1ede422aa402d373cb289702'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:50.974540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1063/2113] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:50 2022] DELETE /identity/v3/users/be87a99b1ede422aa402d373cb289702 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:51.239909 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64728df3-d687-4b03-acba-c9452134a106 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:51.241423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-64728df3-d687-4b03-acba-c9452134a106 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:51.267677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64728df3-d687-4b03-acba-c9452134a106 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:51.283506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:51.287755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:51.351329 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:51.352085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.352194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.352469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] PATH_INFO: `/v3/projects/30703798604e4e8f9ba00a8df2fd4dd6` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.353616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] RBAC: Authorizing `identity:delete_project(project_id=30703798604e4e8f9ba00a8df2fd4dd6)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:51.355558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:51.365941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa3fd4a-1c36-4fae-8c3f-4b804d270510 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:51.372514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:51.373150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.373581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.373865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] PATH_INFO: `/v3/projects/5f9065bedcfd4e749d70335a50a0a5f5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.375021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f9065bedcfd4e749d70335a50a0a5f5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:51.377128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:51.405034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:51.412789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30703798604e4e8f9ba00a8df2fd4dd6', 'request_id': 'req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.421812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30703798604e4e8f9ba00a8df2fd4dd6', 'request_id': 'req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.430343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30703798604e4e8f9ba00a8df2fd4dd6', 'request_id': 'req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.433653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f9065bedcfd4e749d70335a50a0a5f5', 'request_id': 'req-6548c005-588e-46ab-b41f-0e3ce823fec6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.441685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f9065bedcfd4e749d70335a50a0a5f5', 'request_id': 'req-6548c005-588e-46ab-b41f-0e3ce823fec6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.444309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] The token cache is being invalidate because project 30703798604e4e8f9ba00a8df2fd4dd6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:51.444608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.445272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.447571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1051/2114] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:51 2022] DELETE /identity/v3/projects/30703798604e4e8f9ba00a8df2fd4dd6 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:51.448732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f9065bedcfd4e749d70335a50a0a5f5', 'request_id': 'req-6548c005-588e-46ab-b41f-0e3ce823fec6'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.451653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:51.452552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:51.463573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] The token cache is being invalidate because project 5f9065bedcfd4e749d70335a50a0a5f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:51.464115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.464846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6548c005-588e-46ab-b41f-0e3ce823fec6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.466728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1064/2115] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:51 2022] DELETE /identity/v3/projects/5f9065bedcfd4e749d70335a50a0a5f5 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:51.474223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe17304f-ad60-444c-8ff5-37f81497fe7f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.474667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe17304f-ad60-444c-8ff5-37f81497fe7f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.474938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe17304f-ad60-444c-8ff5-37f81497fe7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.496993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fe17304f-ad60-444c-8ff5-37f81497fe7f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:51.498398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fe17304f-ad60-444c-8ff5-37f81497fe7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:51.530510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:51.536029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e84cfe-60b4-4faf-b53d-47f586e1ef7a admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:51.562218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1065/2116] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:51.570529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5886407b-8143-4705-9073-2b3c42c444a6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.570851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5886407b-8143-4705-9073-2b3c42c444a6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.571126 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5886407b-8143-4705-9073-2b3c42c444a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.591572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5886407b-8143-4705-9073-2b3c42c444a6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:51.592323 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:51.592765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5886407b-8143-4705-9073-2b3c42c444a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:51.592944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.593219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.593488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] PATH_INFO: `/v3/users/38b94c23f90348a190b2ddedf6366ea3` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.594623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] RBAC: Authorizing `identity:delete_user(user_id=38b94c23f90348a190b2ddedf6366ea3)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:51.596506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:51.600929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1066/2117] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:51.605244 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5886407b-8143-4705-9073-2b3c42c444a6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:51.614845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5886407b-8143-4705-9073-2b3c42c444a6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:51.619940 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785587d8-ca84-431a-a7aa-126e2a30007b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:51.620529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785587d8-ca84-431a-a7aa-126e2a30007b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.621073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785587d8-ca84-431a-a7aa-126e2a30007b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.621394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785587d8-ca84-431a-a7aa-126e2a30007b admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.622107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785587d8-ca84-431a-a7aa-126e2a30007b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:51.622454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785587d8-ca84-431a-a7aa-126e2a30007b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:51.624146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785587d8-ca84-431a-a7aa-126e2a30007b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:51.640161 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1067/2118] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:55:51.641078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785587d8-ca84-431a-a7aa-126e2a30007b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:51.643310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38b94c23f90348a190b2ddedf6366ea3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.643965 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38b94c23f90348a190b2ddedf6366ea3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.651044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38b94c23f90348a190b2ddedf6366ea3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.651705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38b94c23f90348a190b2ddedf6366ea3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.651812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785587d8-ca84-431a-a7aa-126e2a30007b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:51.656275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ec5956-e12b-4218-a345-06abee2c9f83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:51.656875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ec5956-e12b-4218-a345-06abee2c9f83 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.657149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ec5956-e12b-4218-a345-06abee2c9f83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.657414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ec5956-e12b-4218-a345-06abee2c9f83 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.658047 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ec5956-e12b-4218-a345-06abee2c9f83 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:51.659920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38b94c23f90348a190b2ddedf6366ea3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:51.660614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ec5956-e12b-4218-a345-06abee2c9f83 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:51.671663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:51.679926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1052/2119] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:51 2022] DELETE /identity/v3/users/38b94c23f90348a190b2ddedf6366ea3 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:51.692697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1068/2120] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:51 2022] POST /identity/v3/projects => generated 381 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:51.693888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:51.695303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:51.765022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948cdd97-aac2-4dff-bae0-63b451ec4cbc admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:51.823560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0908f59-32aa-4a33-83d8-10b03d85ce14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:51.824314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0908f59-32aa-4a33-83d8-10b03d85ce14 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.824612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0908f59-32aa-4a33-83d8-10b03d85ce14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.824883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0908f59-32aa-4a33-83d8-10b03d85ce14 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.825510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0908f59-32aa-4a33-83d8-10b03d85ce14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:51.827517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0908f59-32aa-4a33-83d8-10b03d85ce14 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:51.852275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1053/2121] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:51 2022] POST /identity/v3/users => generated 291 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:51.856929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ec5956-e12b-4218-a345-06abee2c9f83 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:51.868339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ec5956-e12b-4218-a345-06abee2c9f83 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:51.873328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfabef50-04c5-4d68-9b9d-244fe917b9c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:51.873952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabef50-04c5-4d68-9b9d-244fe917b9c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.874262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabef50-04c5-4d68-9b9d-244fe917b9c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.874533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfabef50-04c5-4d68-9b9d-244fe917b9c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.875223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfabef50-04c5-4d68-9b9d-244fe917b9c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:51.876815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfabef50-04c5-4d68-9b9d-244fe917b9c0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:51.887390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1069/2122] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:51.891458 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0908f59-32aa-4a33-83d8-10b03d85ce14 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:51.901263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0908f59-32aa-4a33-83d8-10b03d85ce14 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:51.905983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30624a17-c0ac-4e63-adf1-0924bcf72e2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:51.906679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30624a17-c0ac-4e63-adf1-0924bcf72e2b admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.907066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30624a17-c0ac-4e63-adf1-0924bcf72e2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.907350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30624a17-c0ac-4e63-adf1-0924bcf72e2b admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.908039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30624a17-c0ac-4e63-adf1-0924bcf72e2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:51.909663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30624a17-c0ac-4e63-adf1-0924bcf72e2b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:51.919939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1054/2123] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:51 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:51.924040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfabef50-04c5-4d68-9b9d-244fe917b9c0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:51.934898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfabef50-04c5-4d68-9b9d-244fe917b9c0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:51.939551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2ac9a1-3872-4928-a183-dfbc67d47fc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:51.940267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ac9a1-3872-4928-a183-dfbc67d47fc4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.940546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ac9a1-3872-4928-a183-dfbc67d47fc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.940820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2ac9a1-3872-4928-a183-dfbc67d47fc4 admin admin] PATH_INFO: `/v3/projects/af89e28375a042cb8ed0d8d8114ec429/users/29b4c8a09c85409284425a24a47b80ba/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:51.957266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2ac9a1-3872-4928-a183-dfbc67d47fc4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=af89e28375a042cb8ed0d8d8114ec429, user_id=29b4c8a09c85409284425a24a47b80ba, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:51.960117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2ac9a1-3872-4928-a183-dfbc67d47fc4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:51.963818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30624a17-c0ac-4e63-adf1-0924bcf72e2b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:51.965662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-30624a17-c0ac-4e63-adf1-0924bcf72e2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:51.980272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30624a17-c0ac-4e63-adf1-0924bcf72e2b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:51.990820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1070/2124] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:51 2022] PUT /identity/v3/projects/af89e28375a042cb8ed0d8d8114ec429/users/29b4c8a09c85409284425a24a47b80ba/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:51.994565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a0399-b732-400f-81e6-e0da2c8aba9c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:51.994924 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a0399-b732-400f-81e6-e0da2c8aba9c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:51.995272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a0399-b732-400f-81e6-e0da2c8aba9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.026246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f78a0399-b732-400f-81e6-e0da2c8aba9c None None] MFA Rules not processed for user `29b4c8a09c85409284425a24a47b80ba`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:52.027522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f78a0399-b732-400f-81e6-e0da2c8aba9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:52.046687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.047319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.047595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.047870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] PATH_INFO: `/v3/projects/26ecde8251e04541af230eafe62c460b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.048974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26ecde8251e04541af230eafe62c460b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.051458 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.077593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:52.100303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1071/2125] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:51 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:52.105275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a0399-b732-400f-81e6-e0da2c8aba9c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.111193 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26ecde8251e04541af230eafe62c460b', 'request_id': 'req-7df55614-84c3-4925-acc4-719e32e63727'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:52.119463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26ecde8251e04541af230eafe62c460b', 'request_id': 'req-7df55614-84c3-4925-acc4-719e32e63727'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:52.124242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a0399-b732-400f-81e6-e0da2c8aba9c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.127096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26ecde8251e04541af230eafe62c460b', 'request_id': 'req-7df55614-84c3-4925-acc4-719e32e63727'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:52.142771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] The token cache is being invalidate because project 26ecde8251e04541af230eafe62c460b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:52.143894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:52.144867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7df55614-84c3-4925-acc4-719e32e63727 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:52.146739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1055/2126] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:51 2022] DELETE /identity/v3/projects/26ecde8251e04541af230eafe62c460b => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:52.151451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb9ba9a-3295-4369-b431-fcee0f4ea620 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.151801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb9ba9a-3295-4369-b431-fcee0f4ea620 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.152140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb9ba9a-3295-4369-b431-fcee0f4ea620 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.172428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:52.172609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-3eb9ba9a-3295-4369-b431-fcee0f4ea620 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:52.173844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3eb9ba9a-3295-4369-b431-fcee0f4ea620 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:52.185006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1394d23-ae79-42ed-bf38-278e7daa5021 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.185688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1394d23-ae79-42ed-bf38-278e7daa5021 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.186040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1394d23-ae79-42ed-bf38-278e7daa5021 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.186348 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1394d23-ae79-42ed-bf38-278e7daa5021 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.186983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1394d23-ae79-42ed-bf38-278e7daa5021 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.188919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1394d23-ae79-42ed-bf38-278e7daa5021 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.222155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1072/2127] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:52 2022] POST /identity/v3/projects => generated 381 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:52.223113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1394d23-ae79-42ed-bf38-278e7daa5021 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.224849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a1394d23-ae79-42ed-bf38-278e7daa5021 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:52.239736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1056/2128] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:52.249510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141769e4-128f-415e-881c-66da600973f3 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.249967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141769e4-128f-415e-881c-66da600973f3 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.250357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141769e4-128f-415e-881c-66da600973f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.272010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-141769e4-128f-415e-881c-66da600973f3 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:52.273850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-141769e4-128f-415e-881c-66da600973f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:52.301152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1394d23-ae79-42ed-bf38-278e7daa5021 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.306525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f130e9a8-10e5-418f-b079-affd89c27a91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.307220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f130e9a8-10e5-418f-b079-affd89c27a91 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.307773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f130e9a8-10e5-418f-b079-affd89c27a91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.308044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f130e9a8-10e5-418f-b079-affd89c27a91 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.308706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f130e9a8-10e5-418f-b079-affd89c27a91 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.310741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f130e9a8-10e5-418f-b079-affd89c27a91 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.334212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1073/2129] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:52 2022] POST /identity/v3/users => generated 297 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:52.338445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f130e9a8-10e5-418f-b079-affd89c27a91 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.343556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1057/2130] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:52.347718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141769e4-128f-415e-881c-66da600973f3 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.351322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f130e9a8-10e5-418f-b079-affd89c27a91 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.356129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0f8f0a-1475-4657-9f54-7f7db31f89a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.356719 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f8f0a-1475-4657-9f54-7f7db31f89a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.356991 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141769e4-128f-415e-881c-66da600973f3 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.357111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f8f0a-1475-4657-9f54-7f7db31f89a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.357257 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f8f0a-1475-4657-9f54-7f7db31f89a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.357917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0f8f0a-1475-4657-9f54-7f7db31f89a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.359530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0f8f0a-1475-4657-9f54-7f7db31f89a8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.362222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba51f87-346f-4124-baf2-15610d953109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.362962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba51f87-346f-4124-baf2-15610d953109 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.363242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba51f87-346f-4124-baf2-15610d953109 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.363514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba51f87-346f-4124-baf2-15610d953109 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.364176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba51f87-346f-4124-baf2-15610d953109 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:52.364478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba51f87-346f-4124-baf2-15610d953109 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.366048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba51f87-346f-4124-baf2-15610d953109 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.373254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1074/2131] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:52.374212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0f8f0a-1475-4657-9f54-7f7db31f89a8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.379323 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1058/2132] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:55:52.383668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba51f87-346f-4124-baf2-15610d953109 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.384891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0f8f0a-1475-4657-9f54-7f7db31f89a8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.389823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16843355-2073-4ddf-9934-a516ec60490a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.390495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16843355-2073-4ddf-9934-a516ec60490a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.390772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16843355-2073-4ddf-9934-a516ec60490a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.391041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16843355-2073-4ddf-9934-a516ec60490a admin admin] PATH_INFO: `/v3/projects/601ae9f53449484d82614ed615bc4a91/users/0a036d7b2e1f4304a6f2719b21162717/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.394283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba51f87-346f-4124-baf2-15610d953109 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.398830 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6108493c-0c72-4ea6-a1c1-6ced75e1e638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.399410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6108493c-0c72-4ea6-a1c1-6ced75e1e638 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.399707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6108493c-0c72-4ea6-a1c1-6ced75e1e638 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.400027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6108493c-0c72-4ea6-a1c1-6ced75e1e638 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.400650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6108493c-0c72-4ea6-a1c1-6ced75e1e638 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.402640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6108493c-0c72-4ea6-a1c1-6ced75e1e638 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.406448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16843355-2073-4ddf-9934-a516ec60490a admin admin] RBAC: Authorizing `identity:create_grant(project_id=601ae9f53449484d82614ed615bc4a91, user_id=0a036d7b2e1f4304a6f2719b21162717, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.409118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16843355-2073-4ddf-9934-a516ec60490a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.423220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1075/2133] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:52 2022] PUT /identity/v3/projects/601ae9f53449484d82614ed615bc4a91/users/0a036d7b2e1f4304a6f2719b21162717/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:52.428010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16843355-2073-4ddf-9934-a516ec60490a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.434011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1059/2134] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:52 2022] POST /identity/v3/projects => generated 369 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:52.438036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6108493c-0c72-4ea6-a1c1-6ced75e1e638 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.458749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=92,keystone:DELETE=6,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:52.500993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16843355-2073-4ddf-9934-a516ec60490a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.524292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6108493c-0c72-4ea6-a1c1-6ced75e1e638 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.531970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d34f24-20fb-4c8e-9440-793d7265a3fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.538205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d34f24-20fb-4c8e-9440-793d7265a3fe admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.538617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d34f24-20fb-4c8e-9440-793d7265a3fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.538954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d34f24-20fb-4c8e-9440-793d7265a3fe admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.542508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d34f24-20fb-4c8e-9440-793d7265a3fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.544489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d34f24-20fb-4c8e-9440-793d7265a3fe admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.556703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:52.591310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1060/2135] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:52 2022] POST /identity/v3/users => generated 285 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:52.605453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d34f24-20fb-4c8e-9440-793d7265a3fe admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.619961 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0cee2f-cbd2-47fb-871a-bafb2541ad48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.621007 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0cee2f-cbd2-47fb-871a-bafb2541ad48 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.621561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0cee2f-cbd2-47fb-871a-bafb2541ad48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.621844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0cee2f-cbd2-47fb-871a-bafb2541ad48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.622749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0cee2f-cbd2-47fb-871a-bafb2541ad48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.623401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d34f24-20fb-4c8e-9440-793d7265a3fe admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.624707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0cee2f-cbd2-47fb-871a-bafb2541ad48 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.629202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5f5403-221f-48ed-a845-dd3ed8844c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.630023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5f5403-221f-48ed-a845-dd3ed8844c14 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.631421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5f5403-221f-48ed-a845-dd3ed8844c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.631421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5f5403-221f-48ed-a845-dd3ed8844c14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.632214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5f5403-221f-48ed-a845-dd3ed8844c14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.634271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5f5403-221f-48ed-a845-dd3ed8844c14 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.637373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1076/2136] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:52 2022] GET /identity/v3/roles => generated 1420 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:52.642088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0cee2f-cbd2-47fb-871a-bafb2541ad48 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.647797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1061/2137] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:52 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:52.654860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0cee2f-cbd2-47fb-871a-bafb2541ad48 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.660448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5f5403-221f-48ed-a845-dd3ed8844c14 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.662303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c836755-1a8d-41fd-894f-c474d3ab8709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.663693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c836755-1a8d-41fd-894f-c474d3ab8709 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.664046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c836755-1a8d-41fd-894f-c474d3ab8709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.664562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c836755-1a8d-41fd-894f-c474d3ab8709 admin admin] PATH_INFO: `/v3/projects/601ae9f53449484d82614ed615bc4a91/users/0a036d7b2e1f4304a6f2719b21162717/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.667773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c836755-1a8d-41fd-894f-c474d3ab8709 admin admin] RBAC: Authorizing `identity:create_grant(project_id=601ae9f53449484d82614ed615bc4a91, user_id=0a036d7b2e1f4304a6f2719b21162717, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.670471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c836755-1a8d-41fd-894f-c474d3ab8709 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.671313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5f5403-221f-48ed-a845-dd3ed8844c14 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.676690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96349949-aec0-4d9c-8714-abcea8e326b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.677418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96349949-aec0-4d9c-8714-abcea8e326b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.677762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96349949-aec0-4d9c-8714-abcea8e326b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.678162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96349949-aec0-4d9c-8714-abcea8e326b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.678871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96349949-aec0-4d9c-8714-abcea8e326b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.680542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96349949-aec0-4d9c-8714-abcea8e326b2 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.684811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1077/2138] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:52 2022] PUT /identity/v3/projects/601ae9f53449484d82614ed615bc4a91/users/0a036d7b2e1f4304a6f2719b21162717/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:52.704706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1062/2139] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:52 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:52.704706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96349949-aec0-4d9c-8714-abcea8e326b2 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.704997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c016ee18-7c98-4563-b66e-b38e99326d89 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.704997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c016ee18-7c98-4563-b66e-b38e99326d89 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.704997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c016ee18-7c98-4563-b66e-b38e99326d89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.711215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96349949-aec0-4d9c-8714-abcea8e326b2 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.730727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c016ee18-7c98-4563-b66e-b38e99326d89 None None] MFA Rules not processed for user `0a036d7b2e1f4304a6f2719b21162717`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:52.732131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c016ee18-7c98-4563-b66e-b38e99326d89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:52.778912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78678c62-6208-4128-81b3-98ccc7ec161d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.779688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78678c62-6208-4128-81b3-98ccc7ec161d admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.780033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78678c62-6208-4128-81b3-98ccc7ec161d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.780392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78678c62-6208-4128-81b3-98ccc7ec161d admin admin] PATH_INFO: `/v3/projects/101ea502a13246a4a31d6eb69f52df97/users/02c3c69e9fa246e8844f00ce7b4203bf/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.796594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78678c62-6208-4128-81b3-98ccc7ec161d admin admin] RBAC: Authorizing `identity:create_grant(project_id=101ea502a13246a4a31d6eb69f52df97, user_id=02c3c69e9fa246e8844f00ce7b4203bf, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.799387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78678c62-6208-4128-81b3-98ccc7ec161d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.813044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1078/2140] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:52 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:52.813438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1063/2141] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:52 2022] PUT /identity/v3/projects/101ea502a13246a4a31d6eb69f52df97/users/02c3c69e9fa246e8844f00ce7b4203bf/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:52.825337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d500ae3-148e-46ea-b3f8-6994ea3dc0bd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.825657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d500ae3-148e-46ea-b3f8-6994ea3dc0bd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.825787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78678c62-6208-4128-81b3-98ccc7ec161d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.825931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d500ae3-148e-46ea-b3f8-6994ea3dc0bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.827818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-78678c62-6208-4128-81b3-98ccc7ec161d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:52.855305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1d500ae3-148e-46ea-b3f8-6994ea3dc0bd None None] MFA Rules not processed for user `02c3c69e9fa246e8844f00ce7b4203bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:52.856515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1d500ae3-148e-46ea-b3f8-6994ea3dc0bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:52.888136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78678c62-6208-4128-81b3-98ccc7ec161d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:52.892089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dcfa37-7c08-4875-8b29-52ff3a29fd3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:52.892759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dcfa37-7c08-4875-8b29-52ff3a29fd3e service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:52.893661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dcfa37-7c08-4875-8b29-52ff3a29fd3e service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:52.893948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dcfa37-7c08-4875-8b29-52ff3a29fd3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:52.917828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:52.931483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1079/2142] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:52 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:52.935949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d500ae3-148e-46ea-b3f8-6994ea3dc0bd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:52.966534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dcfa37-7c08-4875-8b29-52ff3a29fd3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:52.968166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dcfa37-7c08-4875-8b29-52ff3a29fd3e service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:52.972795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:52.978436 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1064/2143] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:52 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:53.006141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d500ae3-148e-46ea-b3f8-6994ea3dc0bd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:53.015419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d581e16-4ca6-40ed-ab22-5a2883580f59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:53.016792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d581e16-4ca6-40ed-ab22-5a2883580f59 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.016792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d581e16-4ca6-40ed-ab22-5a2883580f59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.016792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d581e16-4ca6-40ed-ab22-5a2883580f59 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.017176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d581e16-4ca6-40ed-ab22-5a2883580f59 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:53.019694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d581e16-4ca6-40ed-ab22-5a2883580f59 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:53.054085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1080/2144] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:52 2022] POST /identity/v3/projects => generated 369 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:53.055452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dcfa37-7c08-4875-8b29-52ff3a29fd3e service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:53.067069 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dcfa37-7c08-4875-8b29-52ff3a29fd3e service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:53.133626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70574103-2990-40c6-b357-3efa9c6372cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:53.134387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70574103-2990-40c6-b357-3efa9c6372cd admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.134724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70574103-2990-40c6-b357-3efa9c6372cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.135271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70574103-2990-40c6-b357-3efa9c6372cd admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.135917 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70574103-2990-40c6-b357-3efa9c6372cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:53.138305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70574103-2990-40c6-b357-3efa9c6372cd admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:53.166150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1065/2145] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:53 2022] POST /identity/v3/users => generated 291 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:53.167369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d581e16-4ca6-40ed-ab22-5a2883580f59 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:53.177839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d581e16-4ca6-40ed-ab22-5a2883580f59 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:53.182950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b4bfff-465c-40d7-af98-1795a88e0172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:53.183559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b4bfff-465c-40d7-af98-1795a88e0172 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.183836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b4bfff-465c-40d7-af98-1795a88e0172 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.184107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b4bfff-465c-40d7-af98-1795a88e0172 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.184749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b4bfff-465c-40d7-af98-1795a88e0172 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:53.186506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b4bfff-465c-40d7-af98-1795a88e0172 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:53.199396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1081/2146] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:53.203951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70574103-2990-40c6-b357-3efa9c6372cd admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:53.214709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70574103-2990-40c6-b357-3efa9c6372cd admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:53.219642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f449f4-01dd-4bf5-a1fd-abcfc7c82c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:53.220273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f449f4-01dd-4bf5-a1fd-abcfc7c82c46 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.220625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f449f4-01dd-4bf5-a1fd-abcfc7c82c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.220844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f449f4-01dd-4bf5-a1fd-abcfc7c82c46 admin admin] PATH_INFO: `/v3/projects/786a9f3301704b11915f03aa4e86f14c/users/2de5cc561b8a456a9dade9e2da91d366/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.237131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f449f4-01dd-4bf5-a1fd-abcfc7c82c46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=786a9f3301704b11915f03aa4e86f14c, user_id=2de5cc561b8a456a9dade9e2da91d366, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:53.240444 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f449f4-01dd-4bf5-a1fd-abcfc7c82c46 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:53.255817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1066/2147] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:53 2022] PUT /identity/v3/projects/786a9f3301704b11915f03aa4e86f14c/users/2de5cc561b8a456a9dade9e2da91d366/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:53.259925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b4bfff-465c-40d7-af98-1795a88e0172 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:53.271704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b4bfff-465c-40d7-af98-1795a88e0172 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:53.313233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f449f4-01dd-4bf5-a1fd-abcfc7c82c46 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:53.315278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b7f449f4-01dd-4bf5-a1fd-abcfc7c82c46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:53.361330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31285e53-6899-4fcd-8653-a3a111543048 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:53.363299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31285e53-6899-4fcd-8653-a3a111543048 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.364003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31285e53-6899-4fcd-8653-a3a111543048 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.364465 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31285e53-6899-4fcd-8653-a3a111543048 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.365447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31285e53-6899-4fcd-8653-a3a111543048 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:53.368419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31285e53-6899-4fcd-8653-a3a111543048 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:53.382838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1082/2148] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:53.383954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31285e53-6899-4fcd-8653-a3a111543048 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:53.406361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f449f4-01dd-4bf5-a1fd-abcfc7c82c46 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:53.429282 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:53.430623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.431042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.431698 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] PATH_INFO: `/v3/users/00bc0881a0ce49d294f480d7d7bb0488` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.434030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] RBAC: Authorizing `identity:delete_user(user_id=00bc0881a0ce49d294f480d7d7bb0488)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:53.436172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:53.462624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31285e53-6899-4fcd-8653-a3a111543048 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:53.469022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3561554-fd31-4b4d-8194-06f968a232a4 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a036d7b2e1f4304a6f2719b21162717', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '601ae9f53449484d82614ed615bc4a91', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:53.469483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=92,keystone:INSERT=4,keystone:DELETE=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:53.469957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3561554-fd31-4b4d-8194-06f968a232a4 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.470412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3561554-fd31-4b4d-8194-06f968a232a4 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.470917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3561554-fd31-4b4d-8194-06f968a232a4 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.472119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3561554-fd31-4b4d-8194-06f968a232a4 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:53.472119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3561554-fd31-4b4d-8194-06f968a232a4 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:53.474667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3561554-fd31-4b4d-8194-06f968a232a4 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:53.493708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1083/2149] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:53.493708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3561554-fd31-4b4d-8194-06f968a232a4 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:53.507171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3561554-fd31-4b4d-8194-06f968a232a4 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:53.516681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6617da7f-bdd4-4546-a261-647d9cd727d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:53.517089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617da7f-bdd4-4546-a261-647d9cd727d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.517364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617da7f-bdd4-4546-a261-647d9cd727d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.517627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6617da7f-bdd4-4546-a261-647d9cd727d5 admin admin] PATH_INFO: `/v3/projects/786a9f3301704b11915f03aa4e86f14c/users/2de5cc561b8a456a9dade9e2da91d366/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.520905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '00bc0881a0ce49d294f480d7d7bb0488'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:53.521037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6617da7f-bdd4-4546-a261-647d9cd727d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=786a9f3301704b11915f03aa4e86f14c, user_id=2de5cc561b8a456a9dade9e2da91d366, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:53.521538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '00bc0881a0ce49d294f480d7d7bb0488'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:53.524200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6617da7f-bdd4-4546-a261-647d9cd727d5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:53.531469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '00bc0881a0ce49d294f480d7d7bb0488'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:53.531469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '00bc0881a0ce49d294f480d7d7bb0488'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:53.539042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1084/2150] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:53 2022] PUT /identity/v3/projects/786a9f3301704b11915f03aa4e86f14c/users/2de5cc561b8a456a9dade9e2da91d366/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:53.548584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6617da7f-bdd4-4546-a261-647d9cd727d5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:53.548757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1855c73a-819a-436a-a2b8-3a638eda0bea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '00bc0881a0ce49d294f480d7d7bb0488'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:53.548867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6617da7f-bdd4-4546-a261-647d9cd727d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:53.566709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1067/2151] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:53 2022] DELETE /identity/v3/users/00bc0881a0ce49d294f480d7d7bb0488 => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:53.566709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca41bc7-797c-4368-9de6-9a8580b91ee2 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.566709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca41bc7-797c-4368-9de6-9a8580b91ee2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.566709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca41bc7-797c-4368-9de6-9a8580b91ee2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.590008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:53.619680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-0ca41bc7-797c-4368-9de6-9a8580b91ee2 None None] MFA Rules not processed for user `2de5cc561b8a456a9dade9e2da91d366`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:53.621646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0ca41bc7-797c-4368-9de6-9a8580b91ee2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:53.673529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6617da7f-bdd4-4546-a261-647d9cd727d5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:53.758105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1068/2152] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:53 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:53.760906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fa9792-cc08-4414-90da-c8e318c72ffe None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.761191 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fa9792-cc08-4414-90da-c8e318c72ffe None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.761464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fa9792-cc08-4414-90da-c8e318c72ffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.810672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-b8fa9792-cc08-4414-90da-c8e318c72ffe None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:53.816233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b8fa9792-cc08-4414-90da-c8e318c72ffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:53.831875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba36db33-5ba5-480d-8dbc-630359bdd8b8 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a036d7b2e1f4304a6f2719b21162717', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '601ae9f53449484d82614ed615bc4a91', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:53.832941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba36db33-5ba5-480d-8dbc-630359bdd8b8 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:53.833722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba36db33-5ba5-480d-8dbc-630359bdd8b8 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:53.834322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba36db33-5ba5-480d-8dbc-630359bdd8b8 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:53.838482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba36db33-5ba5-480d-8dbc-630359bdd8b8 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:53.841886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba36db33-5ba5-480d-8dbc-630359bdd8b8 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:53.891837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1085/2153] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:53 2022] POST /identity/v3/projects => generated 355 bytes in 352 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:53.893766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba36db33-5ba5-480d-8dbc-630359bdd8b8 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:53.896243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ba36db33-5ba5-480d-8dbc-630359bdd8b8 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:53.935795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1069/2154] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:53.939020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fa9792-cc08-4414-90da-c8e318c72ffe None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:53.950540 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fa9792-cc08-4414-90da-c8e318c72ffe None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:54.001386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:54.020864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba36db33-5ba5-480d-8dbc-630359bdd8b8 tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:54.025531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdfdf9c-38b5-400c-9155-c22c24b808c2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:54.030763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdfdf9c-38b5-400c-9155-c22c24b808c2 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:54.031451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdfdf9c-38b5-400c-9155-c22c24b808c2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:54.031882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdfdf9c-38b5-400c-9155-c22c24b808c2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:54.045512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cfa930-8280-402e-b41f-3a88f0d1e74e tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a036d7b2e1f4304a6f2719b21162717', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '601ae9f53449484d82614ed615bc4a91', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:54.046611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cfa930-8280-402e-b41f-3a88f0d1e74e tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:54.046994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cfa930-8280-402e-b41f-3a88f0d1e74e tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:54.047501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cfa930-8280-402e-b41f-3a88f0d1e74e tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:54.048392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cfa930-8280-402e-b41f-3a88f0d1e74e tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:54.048905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cfa930-8280-402e-b41f-3a88f0d1e74e tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:54.051124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cfa930-8280-402e-b41f-3a88f0d1e74e tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:54.067309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1070/2155] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:54.071080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cfa930-8280-402e-b41f-3a88f0d1e74e tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:54.076199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-38cfa930-8280-402e-b41f-3a88f0d1e74e tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:54.133240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdfdf9c-38b5-400c-9155-c22c24b808c2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:54.135185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdfdf9c-38b5-400c-9155-c22c24b808c2 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:54.146707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1086/2156] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:53 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:54.147893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdfdf9c-38b5-400c-9155-c22c24b808c2 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:54.151434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cfa930-8280-402e-b41f-3a88f0d1e74e tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:54.157171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7393dbaa-29eb-424c-ad65-b00a7bb6ac6d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:54.158285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7393dbaa-29eb-424c-ad65-b00a7bb6ac6d service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:54.159037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7393dbaa-29eb-424c-ad65-b00a7bb6ac6d service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:54.159664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7393dbaa-29eb-424c-ad65-b00a7bb6ac6d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:54.167513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdfdf9c-38b5-400c-9155-c22c24b808c2 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:54.173845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a036d7b2e1f4304a6f2719b21162717', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '601ae9f53449484d82614ed615bc4a91', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:54.175061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:54.175346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:54.176044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] PATH_INFO: `/v3/projects/7c351ab10e1f4284854de7daabeadc27` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:54.178731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: Authorizing `identity:delete_project(project_id=7c351ab10e1f4284854de7daabeadc27)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:54.182264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:54.184256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:54.251324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7393dbaa-29eb-424c-ad65-b00a7bb6ac6d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:54.251553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c351ab10e1f4284854de7daabeadc27', 'request_id': 'req-b2a79528-5661-42b2-a93e-5dc4b74fb23b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.253762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7393dbaa-29eb-424c-ad65-b00a7bb6ac6d service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:54.261038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c351ab10e1f4284854de7daabeadc27', 'request_id': 'req-b2a79528-5661-42b2-a93e-5dc4b74fb23b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.264651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1071/2157] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:54.265485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7393dbaa-29eb-424c-ad65-b00a7bb6ac6d service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:54.274473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c351ab10e1f4284854de7daabeadc27', 'request_id': 'req-b2a79528-5661-42b2-a93e-5dc4b74fb23b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.274660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7393dbaa-29eb-424c-ad65-b00a7bb6ac6d service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:54.292811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] The token cache is being invalidate because project 7c351ab10e1f4284854de7daabeadc27 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:54.293099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.293782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.295440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1087/2158] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:54 2022] DELETE /identity/v3/projects/7c351ab10e1f4284854de7daabeadc27 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:54.342553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37be1607-8efb-4d8b-9ca3-f911d70f2d52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:54.343496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37be1607-8efb-4d8b-9ca3-f911d70f2d52 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:54.343879 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37be1607-8efb-4d8b-9ca3-f911d70f2d52 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:54.344254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37be1607-8efb-4d8b-9ca3-f911d70f2d52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:54.346256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-37be1607-8efb-4d8b-9ca3-f911d70f2d52 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:54.378664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:54.417197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37be1607-8efb-4d8b-9ca3-f911d70f2d52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:54.418986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37be1607-8efb-4d8b-9ca3-f911d70f2d52 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:54.438206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1072/2159] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:54 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:54.580415 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:54.580415 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:54.643032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37be1607-8efb-4d8b-9ca3-f911d70f2d52 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:54.644730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-37be1607-8efb-4d8b-9ca3-f911d70f2d52 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:54.671483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a79528-5661-42b2-a93e-5dc4b74fb23b tempest-VolumeQuotaClassesTest-1640965958 tempest-VolumeQuotaClassesTest-1640965958-project-admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:54.677579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:54.678314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:54.678598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:54.678927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] PATH_INFO: `/v3/projects/4cefe7c3f4a84ba1952df6e210278d78` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:54.680149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4cefe7c3f4a84ba1952df6e210278d78)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:54.682106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:54.737228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37be1607-8efb-4d8b-9ca3-f911d70f2d52 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:54.753551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4cefe7c3f4a84ba1952df6e210278d78', 'request_id': 'req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.770226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4cefe7c3f4a84ba1952df6e210278d78', 'request_id': 'req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.781364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4cefe7c3f4a84ba1952df6e210278d78', 'request_id': 'req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.796350 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] The token cache is being invalidate because project 4cefe7c3f4a84ba1952df6e210278d78 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:54.796744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.797557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.800092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1088/2160] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:54 2022] DELETE /identity/v3/projects/4cefe7c3f4a84ba1952df6e210278d78 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:54.805040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:54.806888 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:54.819601 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:54.820443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:54.820723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:54.820996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] PATH_INFO: `/v3/users/29b4c8a09c85409284425a24a47b80ba` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:54.822648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=29b4c8a09c85409284425a24a47b80ba)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:54.824759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:54.842395 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:54.881294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29b4c8a09c85409284425a24a47b80ba'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.881961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29b4c8a09c85409284425a24a47b80ba'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.890226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29b4c8a09c85409284425a24a47b80ba'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.890895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29b4c8a09c85409284425a24a47b80ba'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.896972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5220dd12-8a29-4ef8-9308-14e04cd9e4a9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:54.898399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f60634c8-668a-4734-81ed-6dab7f4af4a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29b4c8a09c85409284425a24a47b80ba'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.902539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:54.903174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:54.903453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:54.903724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] PATH_INFO: `/v3/users/631464266d86495aa9e0ad85a32a80fb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:54.904876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=631464266d86495aa9e0ad85a32a80fb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:54.906930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:54.910263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:54.919756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1073/2161] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:54 2022] DELETE /identity/v3/users/29b4c8a09c85409284425a24a47b80ba => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:54.924592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97ec5f1-18f2-44e5-83fa-4495569a87b4 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:54.925355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97ec5f1-18f2-44e5-83fa-4495569a87b4 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:54.926018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97ec5f1-18f2-44e5-83fa-4495569a87b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:54.946803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f97ec5f1-18f2-44e5-83fa-4495569a87b4 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:54.948084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f97ec5f1-18f2-44e5-83fa-4495569a87b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:54.963451 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '631464266d86495aa9e0ad85a32a80fb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.970789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '631464266d86495aa9e0ad85a32a80fb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.971569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '631464266d86495aa9e0ad85a32a80fb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.972342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '631464266d86495aa9e0ad85a32a80fb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:54.981692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '631464266d86495aa9e0ad85a32a80fb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.004073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1089/2162] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:54 2022] DELETE /identity/v3/users/631464266d86495aa9e0ad85a32a80fb => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:55.016233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1074/2163] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:55.041611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:55.046872 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:55.140406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cfdbbe-6a78-49eb-bdfc-9192b74081d2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:55.148008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c731d4f5-a1a9-4d8a-a55d-97fac5e42a5e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:55.148636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c731d4f5-a1a9-4d8a-a55d-97fac5e42a5e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:55.148916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c731d4f5-a1a9-4d8a-a55d-97fac5e42a5e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:55.149189 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c731d4f5-a1a9-4d8a-a55d-97fac5e42a5e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:55.241735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:55.261776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c731d4f5-a1a9-4d8a-a55d-97fac5e42a5e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:55.263351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c731d4f5-a1a9-4d8a-a55d-97fac5e42a5e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:55.273593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1090/2164] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:55.354834 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97ec5f1-18f2-44e5-83fa-4495569a87b4 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:55.359239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f97ec5f1-18f2-44e5-83fa-4495569a87b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:55.374753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97ec5f1-18f2-44e5-83fa-4495569a87b4 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:55.380645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:55.381363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:55.381714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:55.382090 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] PATH_INFO: `/v3/projects/b1414ab312fe47e88c718e1b8d6bfb04` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:55.383358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1414ab312fe47e88c718e1b8d6bfb04)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:55.385549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:55.442723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1414ab312fe47e88c718e1b8d6bfb04', 'request_id': 'req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.452241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1414ab312fe47e88c718e1b8d6bfb04', 'request_id': 'req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.472451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1414ab312fe47e88c718e1b8d6bfb04', 'request_id': 'req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.487312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] The token cache is being invalidate because project b1414ab312fe47e88c718e1b8d6bfb04 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:55.487681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.488422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7f8f1e39-caae-4c0e-b547-4bbccc4a58b4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.490202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1075/2165] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:55 2022] DELETE /identity/v3/projects/b1414ab312fe47e88c718e1b8d6bfb04 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:55.494912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e2cfd6-32ba-4a42-951a-bf15dbc4e7d0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:55.495419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e2cfd6-32ba-4a42-951a-bf15dbc4e7d0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:55.496000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e2cfd6-32ba-4a42-951a-bf15dbc4e7d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:55.523879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-99e2cfd6-32ba-4a42-951a-bf15dbc4e7d0 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:55.533742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-99e2cfd6-32ba-4a42-951a-bf15dbc4e7d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:55.603281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1091/2166] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:55.611312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8055ff6b-5cea-4920-a9c5-149da5ddf45c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:55.612097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8055ff6b-5cea-4920-a9c5-149da5ddf45c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:55.612368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8055ff6b-5cea-4920-a9c5-149da5ddf45c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:55.621652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e2cfd6-32ba-4a42-951a-bf15dbc4e7d0 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:55.623085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-99e2cfd6-32ba-4a42-951a-bf15dbc4e7d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:55.638960 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e2cfd6-32ba-4a42-951a-bf15dbc4e7d0 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:55.645762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:55.650182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:55.650182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:55.650392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] PATH_INFO: `/v3/projects/af89e28375a042cb8ed0d8d8114ec429` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:55.651649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] RBAC: Authorizing `identity:delete_project(project_id=af89e28375a042cb8ed0d8d8114ec429)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:55.652729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8055ff6b-5cea-4920-a9c5-149da5ddf45c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:55.653740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:55.654654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8055ff6b-5cea-4920-a9c5-149da5ddf45c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:55.679142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1076/2167] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:55.683070 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8055ff6b-5cea-4920-a9c5-149da5ddf45c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:55.698976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8055ff6b-5cea-4920-a9c5-149da5ddf45c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:55.714782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9fe61f-ad89-44c1-b177-68b033fa7f00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:55.715538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9fe61f-ad89-44c1-b177-68b033fa7f00 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:55.715828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9fe61f-ad89-44c1-b177-68b033fa7f00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:55.716105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9fe61f-ad89-44c1-b177-68b033fa7f00 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:55.716784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9fe61f-ad89-44c1-b177-68b033fa7f00 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:55.717115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9fe61f-ad89-44c1-b177-68b033fa7f00 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:55.719019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9fe61f-ad89-44c1-b177-68b033fa7f00 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:55.734982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1077/2168] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:55:55.735610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'af89e28375a042cb8ed0d8d8114ec429', 'request_id': 'req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.740590 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9fe61f-ad89-44c1-b177-68b033fa7f00 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:55.745957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'af89e28375a042cb8ed0d8d8114ec429', 'request_id': 'req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.752076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9fe61f-ad89-44c1-b177-68b033fa7f00 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:55.752964 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'af89e28375a042cb8ed0d8d8114ec429', 'request_id': 'req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.764296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] The token cache is being invalidate because project af89e28375a042cb8ed0d8d8114ec429 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:55.764591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.765271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.766935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1092/2169] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:55 2022] DELETE /identity/v3/projects/af89e28375a042cb8ed0d8d8114ec429 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:55.771955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:55.773341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:55.810812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:55.820520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f19dc5-b1d4-4019-a632-2837d8d9a3cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:55.821399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f19dc5-b1d4-4019-a632-2837d8d9a3cd admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:55.822495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f19dc5-b1d4-4019-a632-2837d8d9a3cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:55.823095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f19dc5-b1d4-4019-a632-2837d8d9a3cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:55.824137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f19dc5-b1d4-4019-a632-2837d8d9a3cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:55.826397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f19dc5-b1d4-4019-a632-2837d8d9a3cd admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:55.861402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc0fb2f-a323-4c4a-b81b-51be19ba067e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:55.867726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:55.870099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:55.870099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:55.870099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] PATH_INFO: `/v3/users/0a036d7b2e1f4304a6f2719b21162717` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:55.871672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a036d7b2e1f4304a6f2719b21162717)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:55.876978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1078/2170] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:55 2022] POST /identity/v3/projects => generated 381 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:55.877135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:55.879530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f19dc5-b1d4-4019-a632-2837d8d9a3cd admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:55.882768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-90f19dc5-b1d4-4019-a632-2837d8d9a3cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:55.891444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:55.934443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a036d7b2e1f4304a6f2719b21162717'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.942732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a036d7b2e1f4304a6f2719b21162717'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.943447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a036d7b2e1f4304a6f2719b21162717'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.944077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a036d7b2e1f4304a6f2719b21162717'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.952549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a036d7b2e1f4304a6f2719b21162717'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:55.973734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f19dc5-b1d4-4019-a632-2837d8d9a3cd admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:55.976632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1093/2171] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:55 2022] DELETE /identity/v3/users/0a036d7b2e1f4304a6f2719b21162717 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:55.980962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87a8ad8-0024-47cf-82a8-b15be005ea0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:55.981579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87a8ad8-0024-47cf-82a8-b15be005ea0e admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:55.981857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87a8ad8-0024-47cf-82a8-b15be005ea0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:55.982160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87a8ad8-0024-47cf-82a8-b15be005ea0e admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:55.982797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87a8ad8-0024-47cf-82a8-b15be005ea0e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:55.984956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87a8ad8-0024-47cf-82a8-b15be005ea0e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.008706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1079/2172] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:55 2022] POST /identity/v3/users => generated 291 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:56.012595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.014292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:56.098347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82dfba93-21ad-47de-b8e2-d086adc45d47 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.103605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0818f6-2630-43fa-80e3-f96db298eaad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.104219 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0818f6-2630-43fa-80e3-f96db298eaad admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.104498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0818f6-2630-43fa-80e3-f96db298eaad admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.104773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0818f6-2630-43fa-80e3-f96db298eaad admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.105472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0818f6-2630-43fa-80e3-f96db298eaad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.107134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0818f6-2630-43fa-80e3-f96db298eaad admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.119418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1094/2173] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:56 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:56.124231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87a8ad8-0024-47cf-82a8-b15be005ea0e admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.134437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87a8ad8-0024-47cf-82a8-b15be005ea0e admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.140273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f820af2b-f476-459a-be20-c2e61a7aa1f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.140953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f820af2b-f476-459a-be20-c2e61a7aa1f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.141460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f820af2b-f476-459a-be20-c2e61a7aa1f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.141744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f820af2b-f476-459a-be20-c2e61a7aa1f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.142481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f820af2b-f476-459a-be20-c2e61a7aa1f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.145144 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f820af2b-f476-459a-be20-c2e61a7aa1f9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.165012 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1080/2174] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:56 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:56.171267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0818f6-2630-43fa-80e3-f96db298eaad admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.187746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0818f6-2630-43fa-80e3-f96db298eaad admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.202684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e36f9a-1e0d-4d4f-a43c-6c8d23602a13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.203368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e36f9a-1e0d-4d4f-a43c-6c8d23602a13 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.203647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e36f9a-1e0d-4d4f-a43c-6c8d23602a13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.203918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e36f9a-1e0d-4d4f-a43c-6c8d23602a13 admin admin] PATH_INFO: `/v3/projects/393e367c92664a4ea78b7116d079b970/users/dde0f9a8c8664597b2112012a165fe6c/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.226410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e36f9a-1e0d-4d4f-a43c-6c8d23602a13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=393e367c92664a4ea78b7116d079b970, user_id=dde0f9a8c8664597b2112012a165fe6c, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.231635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e36f9a-1e0d-4d4f-a43c-6c8d23602a13 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.254089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1095/2175] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:56 2022] PUT /identity/v3/projects/393e367c92664a4ea78b7116d079b970/users/dde0f9a8c8664597b2112012a165fe6c/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:56.265662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73996acd-7a63-49b5-a420-521288bdfb94 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.266043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73996acd-7a63-49b5-a420-521288bdfb94 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.266586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73996acd-7a63-49b5-a420-521288bdfb94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.351094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e36f9a-1e0d-4d4f-a43c-6c8d23602a13 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.353303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d5e36f9a-1e0d-4d4f-a43c-6c8d23602a13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:56.358831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-73996acd-7a63-49b5-a420-521288bdfb94 None None] MFA Rules not processed for user `dde0f9a8c8664597b2112012a165fe6c`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:56.360218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-73996acd-7a63-49b5-a420-521288bdfb94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:56.509717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e36f9a-1e0d-4d4f-a43c-6c8d23602a13 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.517314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.519640 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.520052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.520324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] PATH_INFO: `/v3/projects/601ae9f53449484d82614ed615bc4a91` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.521462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] RBAC: Authorizing `identity:delete_project(project_id=601ae9f53449484d82614ed615bc4a91)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.523390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1081/2176] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:56.527893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73996acd-7a63-49b5-a420-521288bdfb94 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.529352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.537075 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:56.540092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73996acd-7a63-49b5-a420-521288bdfb94 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.550951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947c6e21-814e-4b53-ad65-6765e3858278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.551585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947c6e21-814e-4b53-ad65-6765e3858278 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.551867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947c6e21-814e-4b53-ad65-6765e3858278 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.552142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947c6e21-814e-4b53-ad65-6765e3858278 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.552883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947c6e21-814e-4b53-ad65-6765e3858278 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.554888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947c6e21-814e-4b53-ad65-6765e3858278 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.589362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '601ae9f53449484d82614ed615bc4a91', 'request_id': 'req-03c3cff2-5b94-4d58-a7e1-abe89f645103'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:56.590004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1082/2177] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:56 2022] POST /identity/v3/projects => generated 383 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:56.593910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947c6e21-814e-4b53-ad65-6765e3858278 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.601379 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '601ae9f53449484d82614ed615bc4a91', 'request_id': 'req-03c3cff2-5b94-4d58-a7e1-abe89f645103'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:56.608806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '601ae9f53449484d82614ed615bc4a91', 'request_id': 'req-03c3cff2-5b94-4d58-a7e1-abe89f645103'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:56.623326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] The token cache is being invalidate because project 601ae9f53449484d82614ed615bc4a91 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:56.623858 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:56.624660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03c3cff2-5b94-4d58-a7e1-abe89f645103 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:56.629774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1096/2178] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:56 2022] DELETE /identity/v3/projects/601ae9f53449484d82614ed615bc4a91 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:56.631430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ab672-8a63-4d43-8273-1f5d055101ed None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.631807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ab672-8a63-4d43-8273-1f5d055101ed None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.632240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ab672-8a63-4d43-8273-1f5d055101ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.636595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:56.656898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-731ab672-8a63-4d43-8273-1f5d055101ed None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:56.659193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-731ab672-8a63-4d43-8273-1f5d055101ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:56.675011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947c6e21-814e-4b53-ad65-6765e3858278 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.680130 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9a3ce4-eeb7-44a1-8486-99627185dfbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.680872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9a3ce4-eeb7-44a1-8486-99627185dfbc admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.681221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9a3ce4-eeb7-44a1-8486-99627185dfbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.681571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9a3ce4-eeb7-44a1-8486-99627185dfbc admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.682405 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9a3ce4-eeb7-44a1-8486-99627185dfbc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.684609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9a3ce4-eeb7-44a1-8486-99627185dfbc admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.709317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1083/2179] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:56 2022] POST /identity/v3/users => generated 298 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:55:56.713306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9a3ce4-eeb7-44a1-8486-99627185dfbc admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.715401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ea9a3ce4-eeb7-44a1-8486-99627185dfbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:56.728987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9a3ce4-eeb7-44a1-8486-99627185dfbc admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.735832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e50db80-f258-4417-b4cc-1d60de57d230 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.736388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e50db80-f258-4417-b4cc-1d60de57d230 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.736869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e50db80-f258-4417-b4cc-1d60de57d230 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.737287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e50db80-f258-4417-b4cc-1d60de57d230 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.738028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e50db80-f258-4417-b4cc-1d60de57d230 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.739761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e50db80-f258-4417-b4cc-1d60de57d230 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.752819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1097/2180] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:56.756181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3ef4b-2db2-40fb-a36d-ec216b97a16a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.756684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3ef4b-2db2-40fb-a36d-ec216b97a16a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.757156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3ef4b-2db2-40fb-a36d-ec216b97a16a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.763882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1084/2181] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:56 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:56.770718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e50db80-f258-4417-b4cc-1d60de57d230 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.784573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e50db80-f258-4417-b4cc-1d60de57d230 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.786891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fca3ef4b-2db2-40fb-a36d-ec216b97a16a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:56.788121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fca3ef4b-2db2-40fb-a36d-ec216b97a16a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:56.791140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838348b6-04f5-4c89-91b7-3230dc7313d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.792833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838348b6-04f5-4c89-91b7-3230dc7313d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.793120 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838348b6-04f5-4c89-91b7-3230dc7313d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.793409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838348b6-04f5-4c89-91b7-3230dc7313d7 admin admin] PATH_INFO: `/v3/projects/4330af8cec2d41778a7b3d794770506e/users/62e6688fe084436e8759aa892bb45100/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.812597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1098/2182] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:56.813342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838348b6-04f5-4c89-91b7-3230dc7313d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4330af8cec2d41778a7b3d794770506e, user_id=62e6688fe084436e8759aa892bb45100, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.814358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca3ef4b-2db2-40fb-a36d-ec216b97a16a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.816236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838348b6-04f5-4c89-91b7-3230dc7313d7 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.816800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fca3ef4b-2db2-40fb-a36d-ec216b97a16a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:56.833439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca3ef4b-2db2-40fb-a36d-ec216b97a16a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.834679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1085/2183] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:56 2022] PUT /identity/v3/projects/4330af8cec2d41778a7b3d794770506e/users/62e6688fe084436e8759aa892bb45100/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:55:56.839893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838348b6-04f5-4c89-91b7-3230dc7313d7 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.927301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.928436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.928577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.928764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] PATH_INFO: `/v3/users/29f5e2c344cc4b17be1b175e934a9245` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.930158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=29f5e2c344cc4b17be1b175e934a9245)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.932163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.935658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838348b6-04f5-4c89-91b7-3230dc7313d7 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.941256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5bfd78-9383-44a8-bf52-b3930d46b9d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.941866 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5bfd78-9383-44a8-bf52-b3930d46b9d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.942415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5bfd78-9383-44a8-bf52-b3930d46b9d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.942415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5bfd78-9383-44a8-bf52-b3930d46b9d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.943297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5bfd78-9383-44a8-bf52-b3930d46b9d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:56.943600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5bfd78-9383-44a8-bf52-b3930d46b9d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.945194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5bfd78-9383-44a8-bf52-b3930d46b9d4 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.961063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1086/2184] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:56.961063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5bfd78-9383-44a8-bf52-b3930d46b9d4 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:56.963177 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7c5bfd78-9383-44a8-bf52-b3930d46b9d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:56.977406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5bfd78-9383-44a8-bf52-b3930d46b9d4 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:56.979388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:56.983964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:56.984749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:56.984979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:56.985307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] PATH_INFO: `/v3/users/3e7226a987184c1d9dcdc248d0f4991b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:56.986625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e7226a987184c1d9dcdc248d0f4991b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:56.988741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:56.997287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '29f5e2c344cc4b17be1b175e934a9245'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:57.006242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '29f5e2c344cc4b17be1b175e934a9245'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:57.007233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '29f5e2c344cc4b17be1b175e934a9245'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:57.008612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '29f5e2c344cc4b17be1b175e934a9245'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:57.017048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '29f5e2c344cc4b17be1b175e934a9245'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:57.044450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e7226a987184c1d9dcdc248d0f4991b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:57.045320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e7226a987184c1d9dcdc248d0f4991b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:57.051774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1099/2185] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:56 2022] DELETE /identity/v3/users/29f5e2c344cc4b17be1b175e934a9245 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:57.055029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:57.055029 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:57.056643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e7226a987184c1d9dcdc248d0f4991b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:57.057339 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e7226a987184c1d9dcdc248d0f4991b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:57.067132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e7226a987184c1d9dcdc248d0f4991b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:57.075247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=86,keystone:INSERT=5,keystone:DELETE=9 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:57.091614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1087/2186] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:56 2022] DELETE /identity/v3/users/3e7226a987184c1d9dcdc248d0f4991b => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:57.091614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:57.091614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:57.132207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96030902-0666-4cef-ae38-3b68f25914ee admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:57.183715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d63e59-c676-4168-bccd-19b6f2e9e41d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:57.189563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b66a205-21bc-475e-a2e2-ff0a9d5ad078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:57.190847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b66a205-21bc-475e-a2e2-ff0a9d5ad078 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.191129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b66a205-21bc-475e-a2e2-ff0a9d5ad078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.191397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b66a205-21bc-475e-a2e2-ff0a9d5ad078 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.192035 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b66a205-21bc-475e-a2e2-ff0a9d5ad078 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:57.193951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b66a205-21bc-475e-a2e2-ff0a9d5ad078 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:57.238567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce946d3-ea8e-4157-8285-53e9875cb121 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:57.239136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce946d3-ea8e-4157-8285-53e9875cb121 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.239424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce946d3-ea8e-4157-8285-53e9875cb121 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.239680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce946d3-ea8e-4157-8285-53e9875cb121 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.240338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce946d3-ea8e-4157-8285-53e9875cb121 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:57.241991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce946d3-ea8e-4157-8285-53e9875cb121 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:57.245257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1088/2187] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:57 2022] POST /identity/v3/projects => generated 387 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:57.250292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c49ff3-b5e0-4314-aaf6-f6d2f214c22b None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.250577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c49ff3-b5e0-4314-aaf6-f6d2f214c22b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.250845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c49ff3-b5e0-4314-aaf6-f6d2f214c22b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.274314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1100/2188] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:57 2022] GET /identity/v3/roles => generated 1420 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:57.274758 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-71c49ff3-b5e0-4314-aaf6-f6d2f214c22b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:57.276377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a22a3-1dc3-45af-b6b0-7787b243812f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.276515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-71c49ff3-b5e0-4314-aaf6-f6d2f214c22b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:57.277010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a22a3-1dc3-45af-b6b0-7787b243812f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.278084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a22a3-1dc3-45af-b6b0-7787b243812f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.312226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d00a22a3-1dc3-45af-b6b0-7787b243812f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:57.314188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d00a22a3-1dc3-45af-b6b0-7787b243812f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:57.363688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1089/2189] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:57.364509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c49ff3-b5e0-4314-aaf6-f6d2f214c22b None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:57.378263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c49ff3-b5e0-4314-aaf6-f6d2f214c22b None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:57.385964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcd46af-89c3-41f6-abe0-a64a4ac2d1b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:57.386879 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcd46af-89c3-41f6-abe0-a64a4ac2d1b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.387011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcd46af-89c3-41f6-abe0-a64a4ac2d1b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.387399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bcd46af-89c3-41f6-abe0-a64a4ac2d1b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.387888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcd46af-89c3-41f6-abe0-a64a4ac2d1b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:57.389789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bcd46af-89c3-41f6-abe0-a64a4ac2d1b4 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:57.393300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:57.411730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1101/2190] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:57.412610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00a22a3-1dc3-45af-b6b0-7787b243812f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:57.413847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d00a22a3-1dc3-45af-b6b0-7787b243812f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:57.422122 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1090/2191] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:57 2022] POST /identity/v3/users => generated 294 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:57.422771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcd46af-89c3-41f6-abe0-a64a4ac2d1b4 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:57.425201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4bcd46af-89c3-41f6-abe0-a64a4ac2d1b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:57.429549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00a22a3-1dc3-45af-b6b0-7787b243812f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:57.434475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5105b9-937b-4810-b20a-aac342ac24b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:57.435092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5105b9-937b-4810-b20a-aac342ac24b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.435367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5105b9-937b-4810-b20a-aac342ac24b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.435636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5105b9-937b-4810-b20a-aac342ac24b2 admin admin] PATH_INFO: `/v3/projects/4330af8cec2d41778a7b3d794770506e/users/62e6688fe084436e8759aa892bb45100/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.437941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5105b9-937b-4810-b20a-aac342ac24b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4330af8cec2d41778a7b3d794770506e, user_id=62e6688fe084436e8759aa892bb45100, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:57.440846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5105b9-937b-4810-b20a-aac342ac24b2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:57.451006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1102/2192] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:57 2022] PUT /identity/v3/projects/4330af8cec2d41778a7b3d794770506e/users/62e6688fe084436e8759aa892bb45100/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:57.451866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5105b9-937b-4810-b20a-aac342ac24b2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:57.453401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5e5105b9-937b-4810-b20a-aac342ac24b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:57.497178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bcd46af-89c3-41f6-abe0-a64a4ac2d1b4 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:57.506569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:DELETE=1,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:57.524401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5105b9-937b-4810-b20a-aac342ac24b2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:57.528898 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08466bef-1b17-409a-9dee-abec0ce6a870 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:57.529467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08466bef-1b17-409a-9dee-abec0ce6a870 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.529776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08466bef-1b17-409a-9dee-abec0ce6a870 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.529995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08466bef-1b17-409a-9dee-abec0ce6a870 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.560687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef053642-9278-434c-9676-9b5f2d27b274 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:57.561550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef053642-9278-434c-9676-9b5f2d27b274 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.562047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef053642-9278-434c-9676-9b5f2d27b274 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.562556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef053642-9278-434c-9676-9b5f2d27b274 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.626433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08466bef-1b17-409a-9dee-abec0ce6a870 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:57.627676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08466bef-1b17-409a-9dee-abec0ce6a870 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:57.643521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef053642-9278-434c-9676-9b5f2d27b274 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:57.644355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1103/2193] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:57.645872 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08466bef-1b17-409a-9dee-abec0ce6a870 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:57.647308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef053642-9278-434c-9676-9b5f2d27b274 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:57.657494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08466bef-1b17-409a-9dee-abec0ce6a870 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:57.658317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1091/2194] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:57.660682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ad57a2-9549-459e-bf2a-fcfacdd6a366 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.660682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ad57a2-9549-459e-bf2a-fcfacdd6a366 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.660880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ad57a2-9549-459e-bf2a-fcfacdd6a366 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.663691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea9fbe-f61c-4753-ba09-af35e994db0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:57.664318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea9fbe-f61c-4753-ba09-af35e994db0c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.664575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea9fbe-f61c-4753-ba09-af35e994db0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.664848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccea9fbe-f61c-4753-ba09-af35e994db0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.665479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccea9fbe-f61c-4753-ba09-af35e994db0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:57.667230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccea9fbe-f61c-4753-ba09-af35e994db0c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:57.684798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1104/2195] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:57 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:57.686077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea9fbe-f61c-4753-ba09-af35e994db0c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:57.692343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-d5ad57a2-9549-459e-bf2a-fcfacdd6a366 None None] MFA Rules not processed for user `62e6688fe084436e8759aa892bb45100`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:57.693640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d5ad57a2-9549-459e-bf2a-fcfacdd6a366 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:57.699313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccea9fbe-f61c-4753-ba09-af35e994db0c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:57.707489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0385a6a-234a-4c3c-a1e7-850ee32da3eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:57.708519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0385a6a-234a-4c3c-a1e7-850ee32da3eb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.708800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0385a6a-234a-4c3c-a1e7-850ee32da3eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.709070 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0385a6a-234a-4c3c-a1e7-850ee32da3eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.709857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0385a6a-234a-4c3c-a1e7-850ee32da3eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:57.711908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0385a6a-234a-4c3c-a1e7-850ee32da3eb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:57.731693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1105/2196] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:57 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:57.734727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0385a6a-234a-4c3c-a1e7-850ee32da3eb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:57.756462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0385a6a-234a-4c3c-a1e7-850ee32da3eb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:57.765932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b014351-5ecb-4651-a25f-bcdd94e44360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:57.766796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b014351-5ecb-4651-a25f-bcdd94e44360 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.767313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b014351-5ecb-4651-a25f-bcdd94e44360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.767671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b014351-5ecb-4651-a25f-bcdd94e44360 admin admin] PATH_INFO: `/v3/projects/11099dfe213f4961a914e1314349769c/users/899fef79a0fd43fb944ddd17d166a551/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.800474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b014351-5ecb-4651-a25f-bcdd94e44360 admin admin] RBAC: Authorizing `identity:create_grant(project_id=11099dfe213f4961a914e1314349769c, user_id=899fef79a0fd43fb944ddd17d166a551, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:57.806554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b014351-5ecb-4651-a25f-bcdd94e44360 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:57.844540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1106/2197] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:57 2022] PUT /identity/v3/projects/11099dfe213f4961a914e1314349769c/users/899fef79a0fd43fb944ddd17d166a551/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:57.846881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94d8c72-5686-408e-8888-3c9ac0d46ffb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:57.847169 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94d8c72-5686-408e-8888-3c9ac0d46ffb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:57.847446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94d8c72-5686-408e-8888-3c9ac0d46ffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:57.848705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1092/2198] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:57 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:57.861401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ad57a2-9549-459e-bf2a-fcfacdd6a366 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:57.866875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d5ad57a2-9549-459e-bf2a-fcfacdd6a366 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:57.911130 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c94d8c72-5686-408e-8888-3c9ac0d46ffb None None] MFA Rules not processed for user `899fef79a0fd43fb944ddd17d166a551`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:57.913145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c94d8c72-5686-408e-8888-3c9ac0d46ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:57.960798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:57.997216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:58.011621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ad57a2-9549-459e-bf2a-fcfacdd6a366 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.066982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1107/2199] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:57 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:55:58.069681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94d8c72-5686-408e-8888-3c9ac0d46ffb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.129171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4915bb55-2250-43f4-a894-337be68b56c2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.129787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4915bb55-2250-43f4-a894-337be68b56c2 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.130102 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4915bb55-2250-43f4-a894-337be68b56c2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.130538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4915bb55-2250-43f4-a894-337be68b56c2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.192275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94d8c72-5686-408e-8888-3c9ac0d46ffb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.198659 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70233913-e852-4035-abb2-2f4c2b9cdfb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.199340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70233913-e852-4035-abb2-2f4c2b9cdfb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.201369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70233913-e852-4035-abb2-2f4c2b9cdfb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.201369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70233913-e852-4035-abb2-2f4c2b9cdfb4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.201369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70233913-e852-4035-abb2-2f4c2b9cdfb4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.202957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70233913-e852-4035-abb2-2f4c2b9cdfb4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.213048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4915bb55-2250-43f4-a894-337be68b56c2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.214673 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4915bb55-2250-43f4-a894-337be68b56c2 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.228714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1093/2200] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:57 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:55:58.239520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1108/2201] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:58 2022] POST /identity/v3/projects => generated 387 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:58.243499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4915bb55-2250-43f4-a894-337be68b56c2 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.262145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4915bb55-2250-43f4-a894-337be68b56c2 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.336860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70233913-e852-4035-abb2-2f4c2b9cdfb4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.364496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7d314d-3ec0-4818-a8f7-a6c5ba4c8ca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.365374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7d314d-3ec0-4818-a8f7-a6c5ba4c8ca6 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.365737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7d314d-3ec0-4818-a8f7-a6c5ba4c8ca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.366136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7d314d-3ec0-4818-a8f7-a6c5ba4c8ca6 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.368234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7d314d-3ec0-4818-a8f7-a6c5ba4c8ca6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.372784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7d314d-3ec0-4818-a8f7-a6c5ba4c8ca6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.411082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1094/2202] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:58 2022] POST /identity/v3/users => generated 300 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:58.417145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70233913-e852-4035-abb2-2f4c2b9cdfb4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.420956 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae03dea-1d01-4436-85cc-ec66e75f44a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.421697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae03dea-1d01-4436-85cc-ec66e75f44a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.422044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae03dea-1d01-4436-85cc-ec66e75f44a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.422449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae03dea-1d01-4436-85cc-ec66e75f44a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.424355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7d314d-3ec0-4818-a8f7-a6c5ba4c8ca6 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.429399 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7ae03dea-1d01-4436-85cc-ec66e75f44a7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:58.444432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7d314d-3ec0-4818-a8f7-a6c5ba4c8ca6 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.450564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4305eae9-606a-401f-922e-53119832944e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.451229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4305eae9-606a-401f-922e-53119832944e admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.451509 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4305eae9-606a-401f-922e-53119832944e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.451778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4305eae9-606a-401f-922e-53119832944e admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.452418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4305eae9-606a-401f-922e-53119832944e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.454008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4305eae9-606a-401f-922e-53119832944e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.475377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1095/2203] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:58 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:55:58.479079 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4305eae9-606a-401f-922e-53119832944e admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.484828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4305eae9-606a-401f-922e-53119832944e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:58.498883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4305eae9-606a-401f-922e-53119832944e admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.507876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.508591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.508943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.509795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] PATH_INFO: `/v3/projects/0e5ff02c2cb6476abf55812f209afc9f` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.511484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e5ff02c2cb6476abf55812f209afc9f)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.512006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae03dea-1d01-4436-85cc-ec66e75f44a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.513395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.514020 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae03dea-1d01-4436-85cc-ec66e75f44a7 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.530356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1109/2204] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:58 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:58.532488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae03dea-1d01-4436-85cc-ec66e75f44a7 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.534697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7ae03dea-1d01-4436-85cc-ec66e75f44a7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:58.563165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae03dea-1d01-4436-85cc-ec66e75f44a7 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.563283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:INSERT=3,keystone:DELETE=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:58.570647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.571383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.571637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.572173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] PATH_INFO: `/v3/users/02c3c69e9fa246e8844f00ce7b4203bf` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.573652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=02c3c69e9fa246e8844f00ce7b4203bf)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.575806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.584435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:58.586621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e5ff02c2cb6476abf55812f209afc9f', 'request_id': 'req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.595123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e5ff02c2cb6476abf55812f209afc9f', 'request_id': 'req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.603839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e5ff02c2cb6476abf55812f209afc9f', 'request_id': 'req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.618817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] The token cache is being invalidate because project 0e5ff02c2cb6476abf55812f209afc9f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:58.619100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.620324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.622245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1096/2205] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:58 2022] DELETE /identity/v3/projects/0e5ff02c2cb6476abf55812f209afc9f => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:58.622956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.624522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:58.646814 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02c3c69e9fa246e8844f00ce7b4203bf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.658730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02c3c69e9fa246e8844f00ce7b4203bf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.658730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02c3c69e9fa246e8844f00ce7b4203bf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.659323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02c3c69e9fa246e8844f00ce7b4203bf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.669712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02c3c69e9fa246e8844f00ce7b4203bf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.696299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1110/2206] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:58 2022] DELETE /identity/v3/users/02c3c69e9fa246e8844f00ce7b4203bf => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:58.697776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.700261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:58.700707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c7a678-f31f-486f-9ce8-e59e36ecfb0b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.782960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448bdd1d-c237-4997-abc5-718871cd232c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.785188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b76660-b5f7-4215-a718-4d55907065c3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.790419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.790676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448bdd1d-c237-4997-abc5-718871cd232c admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.791060 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.791339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.791613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] PATH_INFO: `/v3/projects/c2da67bdb084426ea652d7ff328c42bb` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.791882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448bdd1d-c237-4997-abc5-718871cd232c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.792386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448bdd1d-c237-4997-abc5-718871cd232c admin admin] PATH_INFO: `/v3/projects/57fa5bd9069d456db257e6ac997751ae/users/fa2478694bd141b597d2307a21d21d11/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.792757 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c2da67bdb084426ea652d7ff328c42bb)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.794830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.817093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448bdd1d-c237-4997-abc5-718871cd232c admin admin] RBAC: Authorizing `identity:create_grant(project_id=57fa5bd9069d456db257e6ac997751ae, user_id=fa2478694bd141b597d2307a21d21d11, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.820274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448bdd1d-c237-4997-abc5-718871cd232c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.839425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1097/2207] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:58 2022] PUT /identity/v3/projects/57fa5bd9069d456db257e6ac997751ae/users/fa2478694bd141b597d2307a21d21d11/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:58.839425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a41400-33c5-434b-8b48-47110d798351 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.839425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a41400-33c5-434b-8b48-47110d798351 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.839425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a41400-33c5-434b-8b48-47110d798351 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.856330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c2da67bdb084426ea652d7ff328c42bb', 'request_id': 'req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.860743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-42a41400-33c5-434b-8b48-47110d798351 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:58.862192 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-42a41400-33c5-434b-8b48-47110d798351 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:58.867824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c2da67bdb084426ea652d7ff328c42bb', 'request_id': 'req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.873152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c2da67bdb084426ea652d7ff328c42bb', 'request_id': 'req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.885656 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] The token cache is being invalidate because project c2da67bdb084426ea652d7ff328c42bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:58.885904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.886616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c12b1a66-01e6-4ce2-899a-6c67ec7a9887 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:58.888096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1111/2208] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:58 2022] DELETE /identity/v3/projects/c2da67bdb084426ea652d7ff328c42bb => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:58.889317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde82ae-bac1-447d-8966-71017151656b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.889613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde82ae-bac1-447d-8966-71017151656b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.889871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde82ae-bac1-447d-8966-71017151656b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.909519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:58.910124 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ccde82ae-bac1-447d-8966-71017151656b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:58.911469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ccde82ae-bac1-447d-8966-71017151656b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:58.920186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:58.929929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1098/2209] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:58.930972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a41400-33c5-434b-8b48-47110d798351 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.932111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-42a41400-33c5-434b-8b48-47110d798351 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:58.944185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a41400-33c5-434b-8b48-47110d798351 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.948839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840fa1cc-815d-4cbb-890e-49abb469e818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.949512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840fa1cc-815d-4cbb-890e-49abb469e818 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.949853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840fa1cc-815d-4cbb-890e-49abb469e818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.950231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840fa1cc-815d-4cbb-890e-49abb469e818 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.950882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840fa1cc-815d-4cbb-890e-49abb469e818 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.952553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840fa1cc-815d-4cbb-890e-49abb469e818 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.967797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1099/2210] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:58.967797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840fa1cc-815d-4cbb-890e-49abb469e818 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.969024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-840fa1cc-815d-4cbb-890e-49abb469e818 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:58.977233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1112/2211] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:58.979019 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccde82ae-bac1-447d-8966-71017151656b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:58.980467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ccde82ae-bac1-447d-8966-71017151656b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:58.981318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840fa1cc-815d-4cbb-890e-49abb469e818 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.985996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.986540 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.986817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.987093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] PATH_INFO: `/v3/users/dde0f9a8c8664597b2112012a165fe6c` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:58.988205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=dde0f9a8c8664597b2112012a165fe6c)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:58.990127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:58.992943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccde82ae-bac1-447d-8966-71017151656b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:58.998115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:58.998731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:58.999031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:58.999298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] PATH_INFO: `/v3/users/68bfec3a83174c63b22fff74bdf45765` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.000407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=68bfec3a83174c63b22fff74bdf45765)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.002907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.047249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dde0f9a8c8664597b2112012a165fe6c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.048634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dde0f9a8c8664597b2112012a165fe6c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.056191 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dde0f9a8c8664597b2112012a165fe6c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.056889 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dde0f9a8c8664597b2112012a165fe6c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.058180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '68bfec3a83174c63b22fff74bdf45765'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.064187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-90a7e6dd-ddbb-4ecb-9162-34af598536fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dde0f9a8c8664597b2112012a165fe6c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.065904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '68bfec3a83174c63b22fff74bdf45765'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.066637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '68bfec3a83174c63b22fff74bdf45765'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.067252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '68bfec3a83174c63b22fff74bdf45765'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.074420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '68bfec3a83174c63b22fff74bdf45765'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.083049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1100/2212] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:58 2022] DELETE /identity/v3/users/dde0f9a8c8664597b2112012a165fe6c => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:59.084439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03b7cc1-ecc5-44a1-ba4d-19afeea55889 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.084731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03b7cc1-ecc5-44a1-ba4d-19afeea55889 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.085011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03b7cc1-ecc5-44a1-ba4d-19afeea55889 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.092333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1113/2213] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:58 2022] DELETE /identity/v3/users/68bfec3a83174c63b22fff74bdf45765 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:59.093232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.094614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.109206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f03b7cc1-ecc5-44a1-ba4d-19afeea55889 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:59.110457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f03b7cc1-ecc5-44a1-ba4d-19afeea55889 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.172225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7963b2-41f5-42a7-af36-82ad54482ee7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.179849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1101/2214] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:59.180833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72162123-1d86-4f96-a8ad-ff2f23e037ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.180986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03b7cc1-ecc5-44a1-ba4d-19afeea55889 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.181831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72162123-1d86-4f96-a8ad-ff2f23e037ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.182173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72162123-1d86-4f96-a8ad-ff2f23e037ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.182657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72162123-1d86-4f96-a8ad-ff2f23e037ba admin admin] PATH_INFO: `/v3/projects/57fa5bd9069d456db257e6ac997751ae/users/fa2478694bd141b597d2307a21d21d11/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.184031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f03b7cc1-ecc5-44a1-ba4d-19afeea55889 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.184691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72162123-1d86-4f96-a8ad-ff2f23e037ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=57fa5bd9069d456db257e6ac997751ae, user_id=fa2478694bd141b597d2307a21d21d11, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.187436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72162123-1d86-4f96-a8ad-ff2f23e037ba admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.199390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1114/2215] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:55:59 2022] PUT /identity/v3/projects/57fa5bd9069d456db257e6ac997751ae/users/fa2478694bd141b597d2307a21d21d11/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:59.204708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96516bf8-5834-48a0-840a-fb327584cc73 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.205319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96516bf8-5834-48a0-840a-fb327584cc73 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.206845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96516bf8-5834-48a0-840a-fb327584cc73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.230069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-96516bf8-5834-48a0-840a-fb327584cc73 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:59.231330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-96516bf8-5834-48a0-840a-fb327584cc73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.245466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:59.258141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:59.262648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03b7cc1-ecc5-44a1-ba4d-19afeea55889 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.309570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1115/2216] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:59.310928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96516bf8-5834-48a0-840a-fb327584cc73 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.321590 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95eddf9-47df-43dd-aeaa-cfb8983f4b29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.322234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95eddf9-47df-43dd-aeaa-cfb8983f4b29 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.322764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95eddf9-47df-43dd-aeaa-cfb8983f4b29 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.323038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95eddf9-47df-43dd-aeaa-cfb8983f4b29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.324049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96516bf8-5834-48a0-840a-fb327584cc73 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.324789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a95eddf9-47df-43dd-aeaa-cfb8983f4b29 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.330418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757e412e-0030-4b68-be72-9924fae41d9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.330418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757e412e-0030-4b68-be72-9924fae41d9b admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.330418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757e412e-0030-4b68-be72-9924fae41d9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.330633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757e412e-0030-4b68-be72-9924fae41d9b admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.331528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757e412e-0030-4b68-be72-9924fae41d9b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:55:59.331528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757e412e-0030-4b68-be72-9924fae41d9b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.333203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757e412e-0030-4b68-be72-9924fae41d9b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.345840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95eddf9-47df-43dd-aeaa-cfb8983f4b29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.347477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95eddf9-47df-43dd-aeaa-cfb8983f4b29 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.349761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1116/2217] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:55:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:55:59.351293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee03127-5d89-4af3-aa4b-81c19d13f60d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.351781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee03127-5d89-4af3-aa4b-81c19d13f60d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.352346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee03127-5d89-4af3-aa4b-81c19d13f60d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.363210 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1102/2218] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:59.364335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95eddf9-47df-43dd-aeaa-cfb8983f4b29 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.379510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95eddf9-47df-43dd-aeaa-cfb8983f4b29 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.390457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5ee03127-5d89-4af3-aa4b-81c19d13f60d None None] MFA Rules not processed for user `fa2478694bd141b597d2307a21d21d11`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:55:59.390626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff018489-6272-44ce-9c5a-2dfb0399e975 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.391142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff018489-6272-44ce-9c5a-2dfb0399e975 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.391434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff018489-6272-44ce-9c5a-2dfb0399e975 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.391688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff018489-6272-44ce-9c5a-2dfb0399e975 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.391975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5ee03127-5d89-4af3-aa4b-81c19d13f60d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.412838 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff018489-6272-44ce-9c5a-2dfb0399e975 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.414667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff018489-6272-44ce-9c5a-2dfb0399e975 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.432390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1103/2219] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:59.443707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff018489-6272-44ce-9c5a-2dfb0399e975 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.483848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff018489-6272-44ce-9c5a-2dfb0399e975 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.503679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e07f95-4161-436b-9597-9e7ac1e51809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.507218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e07f95-4161-436b-9597-9e7ac1e51809 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.508623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e07f95-4161-436b-9597-9e7ac1e51809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.512844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e07f95-4161-436b-9597-9e7ac1e51809 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.514089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e07f95-4161-436b-9597-9e7ac1e51809 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.516286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e07f95-4161-436b-9597-9e7ac1e51809 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.526375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1117/2220] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:55:59.526375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee03127-5d89-4af3-aa4b-81c19d13f60d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.526375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5ee03127-5d89-4af3-aa4b-81c19d13f60d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.544891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee03127-5d89-4af3-aa4b-81c19d13f60d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.565002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.566075 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.566859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.566859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] PATH_INFO: `/v3/projects/9d7581e33ce04f3eba505fdf077e45c0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.570692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d7581e33ce04f3eba505fdf077e45c0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.572721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1104/2221] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:55:59 2022] POST /identity/v3/projects => generated 375 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:59.572837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.573795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e07f95-4161-436b-9597-9e7ac1e51809 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.575634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d4e07f95-4161-436b-9597-9e7ac1e51809 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.659462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d7581e33ce04f3eba505fdf077e45c0', 'request_id': 'req-6fbff630-274d-49ee-9cc7-8d88874da2a9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.669622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e07f95-4161-436b-9597-9e7ac1e51809 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.672849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d7581e33ce04f3eba505fdf077e45c0', 'request_id': 'req-6fbff630-274d-49ee-9cc7-8d88874da2a9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.682197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d7581e33ce04f3eba505fdf077e45c0', 'request_id': 'req-6fbff630-274d-49ee-9cc7-8d88874da2a9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.700203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] The token cache is being invalidate because project 9d7581e33ce04f3eba505fdf077e45c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:55:59.700934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.702379 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:55:59.705532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1118/2222] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:55:59 2022] DELETE /identity/v3/projects/9d7581e33ce04f3eba505fdf077e45c0 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:55:59.706703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.708508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.737159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:59.749726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63d42ff-50b9-4b0b-a086-8d9b8e833f83 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.750427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63d42ff-50b9-4b0b-a086-8d9b8e833f83 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.750694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63d42ff-50b9-4b0b-a086-8d9b8e833f83 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.751161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63d42ff-50b9-4b0b-a086-8d9b8e833f83 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.757204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b63d42ff-50b9-4b0b-a086-8d9b8e833f83 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.804259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbff630-274d-49ee-9cc7-8d88874da2a9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.816230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefab763-6cc0-4aa4-ac8a-ba8883a23993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.822666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:55:59.833206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefab763-6cc0-4aa4-ac8a-ba8883a23993 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.834198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefab763-6cc0-4aa4-ac8a-ba8883a23993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.834640 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefab763-6cc0-4aa4-ac8a-ba8883a23993 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.835826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefab763-6cc0-4aa4-ac8a-ba8883a23993 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.838291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefab763-6cc0-4aa4-ac8a-ba8883a23993 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.871761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63d42ff-50b9-4b0b-a086-8d9b8e833f83 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.873542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63d42ff-50b9-4b0b-a086-8d9b8e833f83 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.874447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1119/2223] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:55:59 2022] POST /identity/v3/users => generated 288 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:55:59.875597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefab763-6cc0-4aa4-ac8a-ba8883a23993 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.878136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-eefab763-6cc0-4aa4-ac8a-ba8883a23993 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:55:59.885183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1105/2224] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:55:59 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:55:59.886024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63d42ff-50b9-4b0b-a086-8d9b8e833f83 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.897767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefab763-6cc0-4aa4-ac8a-ba8883a23993 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.900385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63d42ff-50b9-4b0b-a086-8d9b8e833f83 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.903617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.904583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.905074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.905406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] PATH_INFO: `/v3/users/07655783f09c449b94b1c85ec33fd4f0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.907644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] RBAC: Authorizing `identity:delete_user(user_id=07655783f09c449b94b1c85ec33fd4f0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.911638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.913425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0d3668-0bcb-4f4a-b916-a1fc2407936c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.914490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d3668-0bcb-4f4a-b916-a1fc2407936c admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.914968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d3668-0bcb-4f4a-b916-a1fc2407936c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.915256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0d3668-0bcb-4f4a-b916-a1fc2407936c admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.915915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0d3668-0bcb-4f4a-b916-a1fc2407936c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.917501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0d3668-0bcb-4f4a-b916-a1fc2407936c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:55:59.950483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1106/2225] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:59 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:55:59.951760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0d3668-0bcb-4f4a-b916-a1fc2407936c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:55:59.969802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0d3668-0bcb-4f4a-b916-a1fc2407936c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:55:59.985459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dba6207-1e25-4429-b48b-fdd31300fa48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:55:59.986869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dba6207-1e25-4429-b48b-fdd31300fa48 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:55:59.987833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dba6207-1e25-4429-b48b-fdd31300fa48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:55:59.988132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dba6207-1e25-4429-b48b-fdd31300fa48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:55:59.988787 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dba6207-1e25-4429-b48b-fdd31300fa48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:55:59.992302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dba6207-1e25-4429-b48b-fdd31300fa48 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:00.011556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '07655783f09c449b94b1c85ec33fd4f0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.013290 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1107/2226] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:55:59 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:00.014324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dba6207-1e25-4429-b48b-fdd31300fa48 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:00.016238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5dba6207-1e25-4429-b48b-fdd31300fa48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:00.044205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '07655783f09c449b94b1c85ec33fd4f0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.045392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dba6207-1e25-4429-b48b-fdd31300fa48 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:00.048761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '07655783f09c449b94b1c85ec33fd4f0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.051990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '07655783f09c449b94b1c85ec33fd4f0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.053861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:00.062240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:00.062240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:00.062240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] PATH_INFO: `/v3/projects/101ea502a13246a4a31d6eb69f52df97` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:00.063845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=101ea502a13246a4a31d6eb69f52df97)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:00.066777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:00.067241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '07655783f09c449b94b1c85ec33fd4f0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.091262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1120/2227] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:55:59 2022] DELETE /identity/v3/users/07655783f09c449b94b1c85ec33fd4f0 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:00.103010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:00.105041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:00.153131 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '101ea502a13246a4a31d6eb69f52df97', 'request_id': 'req-24b7dbe0-f7bf-4077-a990-73517a83fcb3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.165169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '101ea502a13246a4a31d6eb69f52df97', 'request_id': 'req-24b7dbe0-f7bf-4077-a990-73517a83fcb3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.180478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '101ea502a13246a4a31d6eb69f52df97', 'request_id': 'req-24b7dbe0-f7bf-4077-a990-73517a83fcb3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.196007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] The token cache is being invalidate because project 101ea502a13246a4a31d6eb69f52df97 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:00.196413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.197881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.198003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b341334c-dd4e-4d23-af6a-dd00fd39cc07 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:00.200731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1108/2228] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:00 2022] DELETE /identity/v3/projects/101ea502a13246a4a31d6eb69f52df97 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:00.201491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:00.203562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:00.226390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:00.278547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea301a5a-d265-4659-8051-3db05b837e28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:00.279213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea301a5a-d265-4659-8051-3db05b837e28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:00.279485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea301a5a-d265-4659-8051-3db05b837e28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:00.279754 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea301a5a-d265-4659-8051-3db05b837e28 admin admin] PATH_INFO: `/v3/projects/dd7f11d7fea44639a879801793f1165b/users/6ee08a1127ee40f78e6a6237da0f5ce8/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:00.309459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b7dbe0-f7bf-4077-a990-73517a83fcb3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:00.315773 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:00.317050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:00.317299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:00.317579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] PATH_INFO: `/v3/projects/393e367c92664a4ea78b7116d079b970` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:00.318888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] RBAC: Authorizing `identity:delete_project(project_id=393e367c92664a4ea78b7116d079b970)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:00.321789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:00.322618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea301a5a-d265-4659-8051-3db05b837e28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd7f11d7fea44639a879801793f1165b, user_id=6ee08a1127ee40f78e6a6237da0f5ce8, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:00.330638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea301a5a-d265-4659-8051-3db05b837e28 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:00.345228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1121/2229] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:00 2022] PUT /identity/v3/projects/dd7f11d7fea44639a879801793f1165b/users/6ee08a1127ee40f78e6a6237da0f5ce8/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:00.346770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea301a5a-d265-4659-8051-3db05b837e28 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:00.348634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ea301a5a-d265-4659-8051-3db05b837e28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:00.396162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:00.410328 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '393e367c92664a4ea78b7116d079b970', 'request_id': 'req-13a63dbc-8166-4b70-a2c8-152e61da973c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.426576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '393e367c92664a4ea78b7116d079b970', 'request_id': 'req-13a63dbc-8166-4b70-a2c8-152e61da973c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.432435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea301a5a-d265-4659-8051-3db05b837e28 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:00.444278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '393e367c92664a4ea78b7116d079b970', 'request_id': 'req-13a63dbc-8166-4b70-a2c8-152e61da973c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.459713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] The token cache is being invalidate because project 393e367c92664a4ea78b7116d079b970 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:00.460004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.460711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-13a63dbc-8166-4b70-a2c8-152e61da973c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.462271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1109/2230] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:00 2022] DELETE /identity/v3/projects/393e367c92664a4ea78b7116d079b970 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:00.467657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0f1e75-d8c2-4814-bfb9-af07df71b07e None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:00.467938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0f1e75-d8c2-4814-bfb9-af07df71b07e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:00.468213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0f1e75-d8c2-4814-bfb9-af07df71b07e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:00.502791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-2d0f1e75-d8c2-4814-bfb9-af07df71b07e None None] MFA Rules not processed for user `6ee08a1127ee40f78e6a6237da0f5ce8`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:00.504001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2d0f1e75-d8c2-4814-bfb9-af07df71b07e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:00.522144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:00.522885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:00.523218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:00.523565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] PATH_INFO: `/v3/users/2de5cc561b8a456a9dade9e2da91d366` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:00.524776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=2de5cc561b8a456a9dade9e2da91d366)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:00.526854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:00.582583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1110/2231] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:00 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:00.583663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0f1e75-d8c2-4814-bfb9-af07df71b07e None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:00.585016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2d0f1e75-d8c2-4814-bfb9-af07df71b07e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:00.597090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2de5cc561b8a456a9dade9e2da91d366'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.600025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0f1e75-d8c2-4814-bfb9-af07df71b07e None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:00.604490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2de5cc561b8a456a9dade9e2da91d366'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.604713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:00.605186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2de5cc561b8a456a9dade9e2da91d366'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.605364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:00.605592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:00.605821 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2de5cc561b8a456a9dade9e2da91d366'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.605964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] PATH_INFO: `/v3/projects/9ea6439ee303450a93934a5799aee0fd` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:00.607565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ea6439ee303450a93934a5799aee0fd)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:00.607706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:00.609455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:00.616090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2de5cc561b8a456a9dade9e2da91d366'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.643071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1122/2232] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:00 2022] DELETE /identity/v3/users/2de5cc561b8a456a9dade9e2da91d366 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:00.644501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:00.646191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:00.671897 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ea6439ee303450a93934a5799aee0fd', 'request_id': 'req-de245bb7-2bf3-4020-8472-5823c3b8b13d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.684807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ea6439ee303450a93934a5799aee0fd', 'request_id': 'req-de245bb7-2bf3-4020-8472-5823c3b8b13d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.692600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ea6439ee303450a93934a5799aee0fd', 'request_id': 'req-de245bb7-2bf3-4020-8472-5823c3b8b13d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.704504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] The token cache is being invalidate because project 9ea6439ee303450a93934a5799aee0fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:00.704791 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.705512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.707067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1111/2233] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:00 2022] DELETE /identity/v3/projects/9ea6439ee303450a93934a5799aee0fd => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:00.708022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:00.709544 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:00.740356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1a6a9f-0344-4ea7-9ae9-07192f7a44fe admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:00.770216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:00.785001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de245bb7-2bf3-4020-8472-5823c3b8b13d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:00.791007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8d8899-6ec1-437f-ba55-6a73a3e9febd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:00.792010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8d8899-6ec1-437f-ba55-6a73a3e9febd service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:00.792625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8d8899-6ec1-437f-ba55-6a73a3e9febd service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:00.792995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8d8899-6ec1-437f-ba55-6a73a3e9febd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:00.795267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6c8d8899-6ec1-437f-ba55-6a73a3e9febd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:00.811561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:00.812476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:00.812752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:00.813023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] PATH_INFO: `/v3/users/62e6688fe084436e8759aa892bb45100` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:00.814306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=62e6688fe084436e8759aa892bb45100)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:00.816957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:00.868768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8d8899-6ec1-437f-ba55-6a73a3e9febd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:00.871136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8d8899-6ec1-437f-ba55-6a73a3e9febd service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:00.883548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1112/2234] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:00 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:00.884434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8d8899-6ec1-437f-ba55-6a73a3e9febd service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:00.886179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6c8d8899-6ec1-437f-ba55-6a73a3e9febd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:00.897027 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '62e6688fe084436e8759aa892bb45100'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.905706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '62e6688fe084436e8759aa892bb45100'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.906700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '62e6688fe084436e8759aa892bb45100'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.907315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '62e6688fe084436e8759aa892bb45100'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.917704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8d8899-6ec1-437f-ba55-6a73a3e9febd service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:00.922165 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-df76cf68-1f63-423d-9519-4aed6261cdb8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '62e6688fe084436e8759aa892bb45100'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:00.925887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:00.926632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:00.927024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:00.927407 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] PATH_INFO: `/v3/users/899fef79a0fd43fb944ddd17d166a551` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:00.929494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=899fef79a0fd43fb944ddd17d166a551)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:00.932387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:00.971403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:00.981692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1123/2235] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:00 2022] DELETE /identity/v3/users/62e6688fe084436e8759aa892bb45100 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:00.983865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3200e536-d49d-4249-985f-0a877bbcbd01 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:00.998556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3200e536-d49d-4249-985f-0a877bbcbd01 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:00.999561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3200e536-d49d-4249-985f-0a877bbcbd01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.037974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '899fef79a0fd43fb944ddd17d166a551'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:01.039340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '899fef79a0fd43fb944ddd17d166a551'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:01.062867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '899fef79a0fd43fb944ddd17d166a551'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:01.063644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '899fef79a0fd43fb944ddd17d166a551'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:01.073505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-ed5a5003-da3f-44be-a55a-01615f666f1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '899fef79a0fd43fb944ddd17d166a551'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:01.077084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3200e536-d49d-4249-985f-0a877bbcbd01 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:01.078532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3200e536-d49d-4249-985f-0a877bbcbd01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:01.098420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1113/2236] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:00 2022] DELETE /identity/v3/users/899fef79a0fd43fb944ddd17d166a551 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:01.103189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76793ff2-f280-4ea0-a644-f18ca7972c62 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.103559 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76793ff2-f280-4ea0-a644-f18ca7972c62 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.103901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76793ff2-f280-4ea0-a644-f18ca7972c62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.132210 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-76793ff2-f280-4ea0-a644-f18ca7972c62 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:01.134332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-76793ff2-f280-4ea0-a644-f18ca7972c62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:01.220515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1124/2237] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:01.227840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3200e536-d49d-4249-985f-0a877bbcbd01 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.230944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3200e536-d49d-4249-985f-0a877bbcbd01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:01.259386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3200e536-d49d-4249-985f-0a877bbcbd01 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.260784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1114/2238] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:01.262262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eaa91c-496f-44fd-92b3-dc34548c678a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.262572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eaa91c-496f-44fd-92b3-dc34548c678a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.263045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eaa91c-496f-44fd-92b3-dc34548c678a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.280584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:01.283549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.283826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.284092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] PATH_INFO: `/v3/projects/786a9f3301704b11915f03aa4e86f14c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.286012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] RBAC: Authorizing `identity:delete_project(project_id=786a9f3301704b11915f03aa4e86f14c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:01.290732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:01.316874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-34eaa91c-496f-44fd-92b3-dc34548c678a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:01.319473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-34eaa91c-496f-44fd-92b3-dc34548c678a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:01.342628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1115/2239] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:01.350530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34eaa91c-496f-44fd-92b3-dc34548c678a None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.352570 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-34eaa91c-496f-44fd-92b3-dc34548c678a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:01.361940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:01.393103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '786a9f3301704b11915f03aa4e86f14c', 'request_id': 'req-d6a923c4-f642-4380-83ca-b7032a32101c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:01.404086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '786a9f3301704b11915f03aa4e86f14c', 'request_id': 'req-d6a923c4-f642-4380-83ca-b7032a32101c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:01.412826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '786a9f3301704b11915f03aa4e86f14c', 'request_id': 'req-d6a923c4-f642-4380-83ca-b7032a32101c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:01.437723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] The token cache is being invalidate because project 786a9f3301704b11915f03aa4e86f14c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:01.438203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:01.441280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:01.444704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1125/2240] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:01 2022] DELETE /identity/v3/projects/786a9f3301704b11915f03aa4e86f14c => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:01.452882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34eaa91c-496f-44fd-92b3-dc34548c678a None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.454533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.456054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:01.533684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a923c4-f642-4380-83ca-b7032a32101c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.534610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f05bfa-fdcb-4fe1-8ad8-808977370d15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:01.535299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f05bfa-fdcb-4fe1-8ad8-808977370d15 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.535584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f05bfa-fdcb-4fe1-8ad8-808977370d15 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.535858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f05bfa-fdcb-4fe1-8ad8-808977370d15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.537623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c7f05bfa-fdcb-4fe1-8ad8-808977370d15 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:01.538710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a631b9b6-9401-437f-af23-6f6ba7de6074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:01.539313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631b9b6-9401-437f-af23-6f6ba7de6074 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.539589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631b9b6-9401-437f-af23-6f6ba7de6074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.539862 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631b9b6-9401-437f-af23-6f6ba7de6074 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.540504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a631b9b6-9401-437f-af23-6f6ba7de6074 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:01.540805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a631b9b6-9401-437f-af23-6f6ba7de6074 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a631b9b6-9401-437f-af23-6f6ba7de6074 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:01.550699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f05bfa-fdcb-4fe1-8ad8-808977370d15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:01.552151 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f05bfa-fdcb-4fe1-8ad8-808977370d15 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:01.556048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1126/2241] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:56:01.560494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec41326-a508-457b-91ef-d39242955a71 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.560882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec41326-a508-457b-91ef-d39242955a71 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.561287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec41326-a508-457b-91ef-d39242955a71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.563004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1116/2242] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:01.563698 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f05bfa-fdcb-4fe1-8ad8-808977370d15 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.573922 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f05bfa-fdcb-4fe1-8ad8-808977370d15 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.579686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d8d560-ba2e-4e51-aac9-9961b26b9d18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:01.580685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d8d560-ba2e-4e51-aac9-9961b26b9d18 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.580685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d8d560-ba2e-4e51-aac9-9961b26b9d18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.580852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d8d560-ba2e-4e51-aac9-9961b26b9d18 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.581461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d8d560-ba2e-4e51-aac9-9961b26b9d18 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:01.583429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d8d560-ba2e-4e51-aac9-9961b26b9d18 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:01.590876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3ec41326-a508-457b-91ef-d39242955a71 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:01.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3ec41326-a508-457b-91ef-d39242955a71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:01.603278 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1127/2243] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:01.613033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ef7bcc-a962-41b3-8570-17473f3142db None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.613368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ef7bcc-a962-41b3-8570-17473f3142db None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.613622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ef7bcc-a962-41b3-8570-17473f3142db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.632011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1117/2244] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:01 2022] POST /identity/v3/projects => generated 373 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:01.635432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d8d560-ba2e-4e51-aac9-9961b26b9d18 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.637922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-31ef7bcc-a962-41b3-8570-17473f3142db None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:01.639036 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-31ef7bcc-a962-41b3-8570-17473f3142db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:01.645819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:01.719612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1128/2245] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:01.725163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ef7bcc-a962-41b3-8570-17473f3142db None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.730816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d8d560-ba2e-4e51-aac9-9961b26b9d18 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.739108 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ef7bcc-a962-41b3-8570-17473f3142db None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.740093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26157bb-b55e-43d0-a0f9-88dab0b0584d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:01.741331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26157bb-b55e-43d0-a0f9-88dab0b0584d admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.742203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26157bb-b55e-43d0-a0f9-88dab0b0584d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.743163 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26157bb-b55e-43d0-a0f9-88dab0b0584d admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.743515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26157bb-b55e-43d0-a0f9-88dab0b0584d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:01.745904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db66c8-8f1b-4da2-a089-76180ae0eeff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:01.746824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26157bb-b55e-43d0-a0f9-88dab0b0584d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:01.747553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db66c8-8f1b-4da2-a089-76180ae0eeff admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.748031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db66c8-8f1b-4da2-a089-76180ae0eeff admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.748474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db66c8-8f1b-4da2-a089-76180ae0eeff admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.750313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30db66c8-8f1b-4da2-a089-76180ae0eeff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:01.750815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30db66c8-8f1b-4da2-a089-76180ae0eeff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:01.753173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30db66c8-8f1b-4da2-a089-76180ae0eeff admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:01.773946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1129/2246] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:56:01.778815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db66c8-8f1b-4da2-a089-76180ae0eeff admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.786920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1118/2247] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:01 2022] POST /identity/v3/users => generated 287 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:56:01.790428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db66c8-8f1b-4da2-a089-76180ae0eeff admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.791165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26157bb-b55e-43d0-a0f9-88dab0b0584d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.796283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f24095-8b98-4f8b-886a-8c7ca965e901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:01.797244 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f24095-8b98-4f8b-886a-8c7ca965e901 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.797244 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f24095-8b98-4f8b-886a-8c7ca965e901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.797481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f24095-8b98-4f8b-886a-8c7ca965e901 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.798139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f24095-8b98-4f8b-886a-8c7ca965e901 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:01.800079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f24095-8b98-4f8b-886a-8c7ca965e901 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:01.805863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26157bb-b55e-43d0-a0f9-88dab0b0584d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.811936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f7515f-9260-4669-b756-c3fbf85e9d67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:01.812665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f7515f-9260-4669-b756-c3fbf85e9d67 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.813046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f7515f-9260-4669-b756-c3fbf85e9d67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.813505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f7515f-9260-4669-b756-c3fbf85e9d67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.814463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f7515f-9260-4669-b756-c3fbf85e9d67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:01.818128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f7515f-9260-4669-b756-c3fbf85e9d67 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:01.837381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1119/2248] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:01 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:01.837576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1130/2249] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:01 2022] POST /identity/v3/projects => generated 385 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:01.838862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f7515f-9260-4669-b756-c3fbf85e9d67 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.838986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f24095-8b98-4f8b-886a-8c7ca965e901 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.851504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f7515f-9260-4669-b756-c3fbf85e9d67 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.869507 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:01.932925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f24095-8b98-4f8b-886a-8c7ca965e901 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.937387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cc3a16-fe4b-4b57-8415-59c243d24ba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:01.938821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cc3a16-fe4b-4b57-8415-59c243d24ba3 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.939314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cc3a16-fe4b-4b57-8415-59c243d24ba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.940008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671e5017-69dc-41bb-bdd2-0d64c8e39686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:01.940491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cc3a16-fe4b-4b57-8415-59c243d24ba3 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.940606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671e5017-69dc-41bb-bdd2-0d64c8e39686 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:01.940870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671e5017-69dc-41bb-bdd2-0d64c8e39686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:01.941307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671e5017-69dc-41bb-bdd2-0d64c8e39686 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:01.941959 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671e5017-69dc-41bb-bdd2-0d64c8e39686 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:01.942990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cc3a16-fe4b-4b57-8415-59c243d24ba3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:01.943766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-671e5017-69dc-41bb-bdd2-0d64c8e39686 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:01.947343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cc3a16-fe4b-4b57-8415-59c243d24ba3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:01.968380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1131/2250] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:01 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:01.968380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671e5017-69dc-41bb-bdd2-0d64c8e39686 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:01.970498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-671e5017-69dc-41bb-bdd2-0d64c8e39686 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:01.990471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671e5017-69dc-41bb-bdd2-0d64c8e39686 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:01.999181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:02.000459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.000841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.001259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] PATH_INFO: `/v3/projects/11099dfe213f4961a914e1314349769c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.003758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] RBAC: Authorizing `identity:delete_project(project_id=11099dfe213f4961a914e1314349769c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:02.006931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1120/2251] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:01 2022] POST /identity/v3/users => generated 293 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:56:02.007064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:02.007699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cc3a16-fe4b-4b57-8415-59c243d24ba3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:02.019322 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cc3a16-fe4b-4b57-8415-59c243d24ba3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:02.026257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c58729a-2f9d-4f19-b76b-efead37db1e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:02.026951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c58729a-2f9d-4f19-b76b-efead37db1e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.027229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c58729a-2f9d-4f19-b76b-efead37db1e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.027741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c58729a-2f9d-4f19-b76b-efead37db1e7 admin admin] PATH_INFO: `/v3/projects/302dd84b0b90485691447badb4536c56/users/bfee7d93eeff4e728abea067c0c42814/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.053582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c58729a-2f9d-4f19-b76b-efead37db1e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=302dd84b0b90485691447badb4536c56, user_id=bfee7d93eeff4e728abea067c0c42814, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:02.057418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c58729a-2f9d-4f19-b76b-efead37db1e7 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:02.070951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '11099dfe213f4961a914e1314349769c', 'request_id': 'req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.077938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1121/2252] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:02 2022] PUT /identity/v3/projects/302dd84b0b90485691447badb4536c56/users/bfee7d93eeff4e728abea067c0c42814/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:02.079288 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c58729a-2f9d-4f19-b76b-efead37db1e7 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:02.083692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '11099dfe213f4961a914e1314349769c', 'request_id': 'req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.103441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '11099dfe213f4961a914e1314349769c', 'request_id': 'req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.109566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:02.122628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] The token cache is being invalidate because project 11099dfe213f4961a914e1314349769c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:02.123100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.123917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e23e9c2-4fd3-4a1c-922f-3c1e79951099 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.127535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1132/2253] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:01 2022] DELETE /identity/v3/projects/11099dfe213f4961a914e1314349769c => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:02.128292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e87d13-f89c-4945-bdbe-89ed7bdfbcba None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.128666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e87d13-f89c-4945-bdbe-89ed7bdfbcba None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.129133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e87d13-f89c-4945-bdbe-89ed7bdfbcba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.152621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-01e87d13-f89c-4945-bdbe-89ed7bdfbcba None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:02.154704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-01e87d13-f89c-4945-bdbe-89ed7bdfbcba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:02.226263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c58729a-2f9d-4f19-b76b-efead37db1e7 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:02.232673 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51579538-db57-4ed5-91e5-70cf3b886bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:02.233502 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51579538-db57-4ed5-91e5-70cf3b886bcc admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.233931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51579538-db57-4ed5-91e5-70cf3b886bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.237442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1133/2254] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:02.237442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e87d13-f89c-4945-bdbe-89ed7bdfbcba None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:02.239635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51579538-db57-4ed5-91e5-70cf3b886bcc admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.240500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51579538-db57-4ed5-91e5-70cf3b886bcc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:02.242501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51579538-db57-4ed5-91e5-70cf3b886bcc admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:02.245834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-01e87d13-f89c-4945-bdbe-89ed7bdfbcba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:02.259404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e87d13-f89c-4945-bdbe-89ed7bdfbcba None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:02.260763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1122/2255] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:02 2022] GET /identity/v3/roles => generated 1420 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:02.261741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51579538-db57-4ed5-91e5-70cf3b886bcc admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:02.264170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-51579538-db57-4ed5-91e5-70cf3b886bcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:02.265011 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:02.265684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.266017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.266455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] PATH_INFO: `/v3/users/fa2478694bd141b597d2307a21d21d11` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.267724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fa2478694bd141b597d2307a21d21d11)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:02.269940 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:02.303113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51579538-db57-4ed5-91e5-70cf3b886bcc admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:02.330543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:02.331961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.332305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.332548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] PATH_INFO: `/v3/projects/4330af8cec2d41778a7b3d794770506e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.335229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] RBAC: Authorizing `identity:delete_project(project_id=4330af8cec2d41778a7b3d794770506e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:02.338272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:02.357864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fa2478694bd141b597d2307a21d21d11'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.373963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fa2478694bd141b597d2307a21d21d11'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.375016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fa2478694bd141b597d2307a21d21d11'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.375777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fa2478694bd141b597d2307a21d21d11'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.391527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fa2478694bd141b597d2307a21d21d11'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.432609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1134/2256] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:02 2022] DELETE /identity/v3/users/fa2478694bd141b597d2307a21d21d11 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:02.433035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:02.434694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:02.439607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4330af8cec2d41778a7b3d794770506e', 'request_id': 'req-d516f579-2614-46a1-8769-08f72345561f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.456150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4330af8cec2d41778a7b3d794770506e', 'request_id': 'req-d516f579-2614-46a1-8769-08f72345561f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.474384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4330af8cec2d41778a7b3d794770506e', 'request_id': 'req-d516f579-2614-46a1-8769-08f72345561f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.516978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] The token cache is being invalidate because project 4330af8cec2d41778a7b3d794770506e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:02.517278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.518521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:02.520134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1123/2257] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:02 2022] DELETE /identity/v3/projects/4330af8cec2d41778a7b3d794770506e => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:02.523556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:02.523556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:02.528448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:02.553361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760c1205-54e5-42dd-a14f-932c0d055c03 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:02.558177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3f4f92-3ecc-4434-95b4-4d7c9dadc37e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:02.558909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3f4f92-3ecc-4434-95b4-4d7c9dadc37e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.559318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3f4f92-3ecc-4434-95b4-4d7c9dadc37e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.559610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3f4f92-3ecc-4434-95b4-4d7c9dadc37e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.562157 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8f3f4f92-3ecc-4434-95b4-4d7c9dadc37e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:02.594511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:02.627650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d516f579-2614-46a1-8769-08f72345561f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:02.668439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3f4f92-3ecc-4434-95b4-4d7c9dadc37e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:02.669900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3f4f92-3ecc-4434-95b4-4d7c9dadc37e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:02.683406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1135/2258] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:02.686261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3b462-848b-4ed4-ae15-da78b5aa8bdc None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.686549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3b462-848b-4ed4-ae15-da78b5aa8bdc None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.686822 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be3b462-848b-4ed4-ae15-da78b5aa8bdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.713966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-1be3b462-848b-4ed4-ae15-da78b5aa8bdc None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:02.715271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1be3b462-848b-4ed4-ae15-da78b5aa8bdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:02.728462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5be5c5f-390d-4a81-9755-a97324a13d72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:02.729245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5be5c5f-390d-4a81-9755-a97324a13d72 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.729676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5be5c5f-390d-4a81-9755-a97324a13d72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.729950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5be5c5f-390d-4a81-9755-a97324a13d72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.731144 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5be5c5f-390d-4a81-9755-a97324a13d72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:02.733065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5be5c5f-390d-4a81-9755-a97324a13d72 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:02.734470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1136/2259] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:02.747456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f80945-33b8-406b-8149-5ffec4cf6d9a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.748104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f80945-33b8-406b-8149-5ffec4cf6d9a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.748104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f80945-33b8-406b-8149-5ffec4cf6d9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.762876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1124/2260] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:02 2022] GET /identity/v3/roles => generated 1420 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:02.771570 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5be5c5f-390d-4a81-9755-a97324a13d72 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:02.799382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5be5c5f-390d-4a81-9755-a97324a13d72 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:02.804857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3f64d3-9b53-4142-b8c0-1bda3be690aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:02.805151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-66f80945-33b8-406b-8149-5ffec4cf6d9a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:02.805849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3f64d3-9b53-4142-b8c0-1bda3be690aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.806224 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3f64d3-9b53-4142-b8c0-1bda3be690aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.806450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3f64d3-9b53-4142-b8c0-1bda3be690aa admin admin] PATH_INFO: `/v3/projects/548b4f8630fd47178cd4ac3dbec806ce/users/c94ec8a20ecb465a80f43a2dd27f53ef/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.807819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-66f80945-33b8-406b-8149-5ffec4cf6d9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:02.820757 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1137/2261] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:02.825446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f80945-33b8-406b-8149-5ffec4cf6d9a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:02.829015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3f64d3-9b53-4142-b8c0-1bda3be690aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=548b4f8630fd47178cd4ac3dbec806ce, user_id=c94ec8a20ecb465a80f43a2dd27f53ef, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:02.835232 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3f64d3-9b53-4142-b8c0-1bda3be690aa admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:02.838009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f80945-33b8-406b-8149-5ffec4cf6d9a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:02.845471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21948914-a3ff-4f19-a181-85edca130945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:02.846580 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21948914-a3ff-4f19-a181-85edca130945 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.846864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21948914-a3ff-4f19-a181-85edca130945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.847252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21948914-a3ff-4f19-a181-85edca130945 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.847805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21948914-a3ff-4f19-a181-85edca130945 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:02.848111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21948914-a3ff-4f19-a181-85edca130945 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:02.849765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21948914-a3ff-4f19-a181-85edca130945 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:02.872922 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1125/2262] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:02 2022] PUT /identity/v3/projects/548b4f8630fd47178cd4ac3dbec806ce/users/c94ec8a20ecb465a80f43a2dd27f53ef/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:02.873141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1138/2263] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:56:02.873228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3f64d3-9b53-4142-b8c0-1bda3be690aa admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:02.877040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d3a540-8c3c-46d7-b238-19fb04440c53 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.877333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d3a540-8c3c-46d7-b238-19fb04440c53 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.877612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d3a540-8c3c-46d7-b238-19fb04440c53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:02.914710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-42d3a540-8c3c-46d7-b238-19fb04440c53 None None] MFA Rules not processed for user `c94ec8a20ecb465a80f43a2dd27f53ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:02.915925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-42d3a540-8c3c-46d7-b238-19fb04440c53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:02.984888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3f64d3-9b53-4142-b8c0-1bda3be690aa admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:02.997815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b523d8-8850-4ac4-abce-01382907efcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:02.998815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b523d8-8850-4ac4-abce-01382907efcc admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:02.999326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b523d8-8850-4ac4-abce-01382907efcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:02.999789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b523d8-8850-4ac4-abce-01382907efcc admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.001558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b523d8-8850-4ac4-abce-01382907efcc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:03.018941 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b523d8-8850-4ac4-abce-01382907efcc admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:03.030205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1139/2264] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:03.034360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d3a540-8c3c-46d7-b238-19fb04440c53 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:03.048007 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d3a540-8c3c-46d7-b238-19fb04440c53 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:03.054973 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9ab3ed-0533-4439-8f06-b02220208ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:03.055632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9ab3ed-0533-4439-8f06-b02220208ce9 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.056019 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9ab3ed-0533-4439-8f06-b02220208ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.056340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9ab3ed-0533-4439-8f06-b02220208ce9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.058202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9ab3ed-0533-4439-8f06-b02220208ce9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:03.065441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9ab3ed-0533-4439-8f06-b02220208ce9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:03.067546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1126/2265] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:02 2022] POST /identity/v3/projects => generated 377 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:03.072087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b523d8-8850-4ac4-abce-01382907efcc admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:03.087997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:03.113629 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1140/2266] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:03 2022] POST /identity/v3/projects => generated 387 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:03.120207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9ab3ed-0533-4439-8f06-b02220208ce9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:03.215907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b523d8-8850-4ac4-abce-01382907efcc admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:03.243162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9ab3ed-0533-4439-8f06-b02220208ce9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:03.253564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ec59c-fdf5-4fac-99bb-77f2272c91b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:03.254782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ec59c-fdf5-4fac-99bb-77f2272c91b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.255071 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ec59c-fdf5-4fac-99bb-77f2272c91b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.255384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ec59c-fdf5-4fac-99bb-77f2272c91b0 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.256056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226ec59c-fdf5-4fac-99bb-77f2272c91b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:03.258448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226ec59c-fdf5-4fac-99bb-77f2272c91b0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:03.270263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:03.299512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1141/2267] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:03 2022] POST /identity/v3/users => generated 300 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:03.301948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ec59c-fdf5-4fac-99bb-77f2272c91b0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:03.304487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-226ec59c-fdf5-4fac-99bb-77f2272c91b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:03.321790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035c802f-c6f8-4085-82ef-44bcebef5824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:03.322689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ec59c-fdf5-4fac-99bb-77f2272c91b0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:03.323363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035c802f-c6f8-4085-82ef-44bcebef5824 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.323795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035c802f-c6f8-4085-82ef-44bcebef5824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.324243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035c802f-c6f8-4085-82ef-44bcebef5824 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.325519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035c802f-c6f8-4085-82ef-44bcebef5824 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:03.328660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:03.328881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035c802f-c6f8-4085-82ef-44bcebef5824 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:03.329308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.329585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.329863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] PATH_INFO: `/v3/projects/57fa5bd9069d456db257e6ac997751ae` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.332266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] RBAC: Authorizing `identity:delete_project(project_id=57fa5bd9069d456db257e6ac997751ae)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:03.337490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:03.377236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1127/2268] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:03 2022] POST /identity/v3/users => generated 289 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:03.381286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035c802f-c6f8-4085-82ef-44bcebef5824 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:03.398766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035c802f-c6f8-4085-82ef-44bcebef5824 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:03.407057 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d64409a-00cc-4f9e-840c-87d9130cfe12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:03.407976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d64409a-00cc-4f9e-840c-87d9130cfe12 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.409385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d64409a-00cc-4f9e-840c-87d9130cfe12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.410159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d64409a-00cc-4f9e-840c-87d9130cfe12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.411245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d64409a-00cc-4f9e-840c-87d9130cfe12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:03.413577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d64409a-00cc-4f9e-840c-87d9130cfe12 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:03.428687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1128/2269] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:03 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:03.430206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d64409a-00cc-4f9e-840c-87d9130cfe12 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:03.451312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57fa5bd9069d456db257e6ac997751ae', 'request_id': 'req-7d81fa67-e043-48df-b1fd-a14587c5ae63'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:03.458932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d64409a-00cc-4f9e-840c-87d9130cfe12 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:03.471630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57fa5bd9069d456db257e6ac997751ae', 'request_id': 'req-7d81fa67-e043-48df-b1fd-a14587c5ae63'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:03.492553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57fa5bd9069d456db257e6ac997751ae', 'request_id': 'req-7d81fa67-e043-48df-b1fd-a14587c5ae63'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:03.520107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] The token cache is being invalidate because project 57fa5bd9069d456db257e6ac997751ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:03.520408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:03.522463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:03.526094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1142/2270] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:03 2022] DELETE /identity/v3/projects/57fa5bd9069d456db257e6ac997751ae => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:03.526855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:03.529745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:03.589517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc05389-06aa-45c3-9c39-8edc529e5ee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:03.590168 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc05389-06aa-45c3-9c39-8edc529e5ee0 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.590468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc05389-06aa-45c3-9c39-8edc529e5ee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.590726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc05389-06aa-45c3-9c39-8edc529e5ee0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.591429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc05389-06aa-45c3-9c39-8edc529e5ee0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:03.593034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc05389-06aa-45c3-9c39-8edc529e5ee0 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:03.604460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1129/2271] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:03 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:03.605547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3612585b-5f48-4047-8689-138709a08a83 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.605837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3612585b-5f48-4047-8689-138709a08a83 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.606133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3612585b-5f48-4047-8689-138709a08a83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.618512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d81fa67-e043-48df-b1fd-a14587c5ae63 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:03.625183 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fbfb77-2f40-4c3c-b874-9a38f32b61fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:03.627446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fbfb77-2f40-4c3c-b874-9a38f32b61fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.627876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fbfb77-2f40-4c3c-b874-9a38f32b61fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.628292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fbfb77-2f40-4c3c-b874-9a38f32b61fd admin admin] PATH_INFO: `/v3/projects/0da0a643eafa4575a16c7820f66e848b/users/99842fc553134bf0a83f25eefcd45073/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.630432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-3612585b-5f48-4047-8689-138709a08a83 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:03.631760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3612585b-5f48-4047-8689-138709a08a83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:03.641551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1130/2272] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:03.643193 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3612585b-5f48-4047-8689-138709a08a83 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:03.644538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3612585b-5f48-4047-8689-138709a08a83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:03.651114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fbfb77-2f40-4c3c-b874-9a38f32b61fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=0da0a643eafa4575a16c7820f66e848b, user_id=99842fc553134bf0a83f25eefcd45073, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:03.656015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fbfb77-2f40-4c3c-b874-9a38f32b61fd admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:03.658368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3612585b-5f48-4047-8689-138709a08a83 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:03.663943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2590d608-61f4-40ba-82c8-41d3f4690c91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:03.664565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2590d608-61f4-40ba-82c8-41d3f4690c91 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.664842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2590d608-61f4-40ba-82c8-41d3f4690c91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.665118 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2590d608-61f4-40ba-82c8-41d3f4690c91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.665759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2590d608-61f4-40ba-82c8-41d3f4690c91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:03.667817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2590d608-61f4-40ba-82c8-41d3f4690c91 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:03.679618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1131/2273] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:03 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:03.681368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad8bd9-4dbd-421d-99f0-b972d40b975f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.681658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad8bd9-4dbd-421d-99f0-b972d40b975f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.681937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adad8bd9-4dbd-421d-99f0-b972d40b975f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.693549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1143/2274] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:03 2022] PUT /identity/v3/projects/0da0a643eafa4575a16c7820f66e848b/users/99842fc553134bf0a83f25eefcd45073/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:03.695142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fbfb77-2f40-4c3c-b874-9a38f32b61fd admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:03.703892 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-adad8bd9-4dbd-421d-99f0-b972d40b975f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:03.705185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-adad8bd9-4dbd-421d-99f0-b972d40b975f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:03.717230 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:03.780084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=5,keystone:SELECT=88,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:03.789390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fbfb77-2f40-4c3c-b874-9a38f32b61fd admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:03.801168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ec2b75-cb9c-433c-9a5d-c21e7f3d0fa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:03.812842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ec2b75-cb9c-433c-9a5d-c21e7f3d0fa2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:03.813484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ec2b75-cb9c-433c-9a5d-c21e7f3d0fa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:03.813946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ec2b75-cb9c-433c-9a5d-c21e7f3d0fa2 admin admin] PATH_INFO: `/v3/projects/08f05a4e059c41f2b889de30b8b78a93/users/aaccd3976d52438e9dc639ce7d0a26f0/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:03.846385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1132/2275] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:03.847340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adad8bd9-4dbd-421d-99f0-b972d40b975f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:03.851106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ec2b75-cb9c-433c-9a5d-c21e7f3d0fa2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=08f05a4e059c41f2b889de30b8b78a93, user_id=aaccd3976d52438e9dc639ce7d0a26f0, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:03.855041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ec2b75-cb9c-433c-9a5d-c21e7f3d0fa2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:03.871759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adad8bd9-4dbd-421d-99f0-b972d40b975f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:03.880459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1144/2276] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:03 2022] PUT /identity/v3/projects/08f05a4e059c41f2b889de30b8b78a93/users/aaccd3976d52438e9dc639ce7d0a26f0/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:03.894088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ec2b75-cb9c-433c-9a5d-c21e7f3d0fa2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:04.004984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874e3c3d-4413-4fe6-bc4d-f7959d82f2c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:04.005803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874e3c3d-4413-4fe6-bc4d-f7959d82f2c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.005923 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874e3c3d-4413-4fe6-bc4d-f7959d82f2c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.006237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874e3c3d-4413-4fe6-bc4d-f7959d82f2c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.006886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874e3c3d-4413-4fe6-bc4d-f7959d82f2c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:04.008690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874e3c3d-4413-4fe6-bc4d-f7959d82f2c1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:04.029068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1133/2277] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:03 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:04.030479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfb947d-2d8c-445a-8697-92a2ab1c1973 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.030774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfb947d-2d8c-445a-8697-92a2ab1c1973 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.031194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfb947d-2d8c-445a-8697-92a2ab1c1973 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.063285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-0dfb947d-2d8c-445a-8697-92a2ab1c1973 None None] MFA Rules not processed for user `aaccd3976d52438e9dc639ce7d0a26f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:04.064681 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0dfb947d-2d8c-445a-8697-92a2ab1c1973 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:04.076519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ec2b75-cb9c-433c-9a5d-c21e7f3d0fa2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:04.084727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86441ac-70f8-4c3f-855c-c220bd1e9532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:04.085400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86441ac-70f8-4c3f-855c-c220bd1e9532 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.085721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86441ac-70f8-4c3f-855c-c220bd1e9532 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.086000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86441ac-70f8-4c3f-855c-c220bd1e9532 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.086691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86441ac-70f8-4c3f-855c-c220bd1e9532 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:04.087229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86441ac-70f8-4c3f-855c-c220bd1e9532 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:04.088613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86441ac-70f8-4c3f-855c-c220bd1e9532 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:04.117015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1145/2278] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:56:04.117859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86441ac-70f8-4c3f-855c-c220bd1e9532 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:04.140336 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86441ac-70f8-4c3f-855c-c220bd1e9532 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:04.162632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a675d375-7176-4361-8eb6-e2ad5bc1c153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:04.165052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a675d375-7176-4361-8eb6-e2ad5bc1c153 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.165641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a675d375-7176-4361-8eb6-e2ad5bc1c153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.165926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a675d375-7176-4361-8eb6-e2ad5bc1c153 admin admin] PATH_INFO: `/v3/projects/0da0a643eafa4575a16c7820f66e848b/users/99842fc553134bf0a83f25eefcd45073/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.169059 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a675d375-7176-4361-8eb6-e2ad5bc1c153 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0da0a643eafa4575a16c7820f66e848b, user_id=99842fc553134bf0a83f25eefcd45073, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:04.171812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a675d375-7176-4361-8eb6-e2ad5bc1c153 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:04.207301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1146/2279] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:04 2022] PUT /identity/v3/projects/0da0a643eafa4575a16c7820f66e848b/users/99842fc553134bf0a83f25eefcd45073/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:04.208163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a675d375-7176-4361-8eb6-e2ad5bc1c153 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:04.230814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1134/2280] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:04 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:04.232576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d127e6df-22ac-4a0e-a37e-8a37c79405e8 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.233071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d127e6df-22ac-4a0e-a37e-8a37c79405e8 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.233547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d127e6df-22ac-4a0e-a37e-8a37c79405e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.270949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-d127e6df-22ac-4a0e-a37e-8a37c79405e8 None None] MFA Rules not processed for user `99842fc553134bf0a83f25eefcd45073`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:04.272580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d127e6df-22ac-4a0e-a37e-8a37c79405e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:04.309030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a675d375-7176-4361-8eb6-e2ad5bc1c153 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:04.315352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71305db0-7e10-4347-90d4-29cdaa6bfca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:04.315986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71305db0-7e10-4347-90d4-29cdaa6bfca1 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.316266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71305db0-7e10-4347-90d4-29cdaa6bfca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.316540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71305db0-7e10-4347-90d4-29cdaa6bfca1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.317185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71305db0-7e10-4347-90d4-29cdaa6bfca1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:04.319641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71305db0-7e10-4347-90d4-29cdaa6bfca1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:04.333598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:04.368360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1147/2281] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:04 2022] POST /identity/v3/projects => generated 383 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:04.369614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71305db0-7e10-4347-90d4-29cdaa6bfca1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:04.373723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-71305db0-7e10-4347-90d4-29cdaa6bfca1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:04.443880 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1135/2282] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:04 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:04.445464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d127e6df-22ac-4a0e-a37e-8a37c79405e8 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:04.457876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d127e6df-22ac-4a0e-a37e-8a37c79405e8 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:04.494829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:04.528861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d71118b-845b-47dc-a99b-a838b5cad1ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:04.529803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71118b-845b-47dc-a99b-a838b5cad1ef admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.530219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71118b-845b-47dc-a99b-a838b5cad1ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.530721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d71118b-845b-47dc-a99b-a838b5cad1ef admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.531556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d71118b-845b-47dc-a99b-a838b5cad1ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:04.534535 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d71118b-845b-47dc-a99b-a838b5cad1ef admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:04.569171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1136/2283] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:04 2022] POST /identity/v3/users => generated 292 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:04.570519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d71118b-845b-47dc-a99b-a838b5cad1ef admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:04.571637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71305db0-7e10-4347-90d4-29cdaa6bfca1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:04.572754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3d71118b-845b-47dc-a99b-a838b5cad1ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:04.581579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4209fd-549a-468e-bd30-d2096de9ded5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc59f80f544cef8b3869e8d37bb341', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:04.582231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4209fd-549a-468e-bd30-d2096de9ded5 service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.582507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4209fd-549a-468e-bd30-d2096de9ded5 service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.582776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4209fd-549a-468e-bd30-d2096de9ded5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.648271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d71118b-845b-47dc-a99b-a838b5cad1ef admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:04.652742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa512ba-5b6d-4abb-a891-dfacd2bb8bdf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:04.653799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa512ba-5b6d-4abb-a891-dfacd2bb8bdf service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.654214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa512ba-5b6d-4abb-a891-dfacd2bb8bdf service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.654604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa512ba-5b6d-4abb-a891-dfacd2bb8bdf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.716604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4209fd-549a-468e-bd30-d2096de9ded5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:04.718187 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed4209fd-549a-468e-bd30-d2096de9ded5 service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:04.742784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1148/2284] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:04 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:04.743819 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4209fd-549a-468e-bd30-d2096de9ded5 service glance] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:04.764381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed4209fd-549a-468e-bd30-d2096de9ded5 service glance] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:04.768253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa512ba-5b6d-4abb-a891-dfacd2bb8bdf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:04.769852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa512ba-5b6d-4abb-a891-dfacd2bb8bdf service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:04.777054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe71d72-9223-47d2-a83c-50eb94049686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:04.777904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe71d72-9223-47d2-a83c-50eb94049686 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.778236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe71d72-9223-47d2-a83c-50eb94049686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.778508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe71d72-9223-47d2-a83c-50eb94049686 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.779417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe71d72-9223-47d2-a83c-50eb94049686 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:04.782536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe71d72-9223-47d2-a83c-50eb94049686 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:04.793875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1137/2285] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:04 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:04.802926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1149/2286] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:04 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:04.806935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa512ba-5b6d-4abb-a891-dfacd2bb8bdf service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:04.812530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:04.823694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa512ba-5b6d-4abb-a891-dfacd2bb8bdf service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:04.842614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c4e451-c984-40d7-a6ed-9578be30abef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:04.843368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c4e451-c984-40d7-a6ed-9578be30abef admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.843713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c4e451-c984-40d7-a6ed-9578be30abef admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.844169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c4e451-c984-40d7-a6ed-9578be30abef admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.846414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c4e451-c984-40d7-a6ed-9578be30abef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:04.849187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c4e451-c984-40d7-a6ed-9578be30abef admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:04.887149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1138/2287] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:04 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:04.887311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe71d72-9223-47d2-a83c-50eb94049686 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:04.929117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe71d72-9223-47d2-a83c-50eb94049686 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:04.946141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfc264e-f5e1-4a17-aa7c-5359b1203fd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:04.952508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfc264e-f5e1-4a17-aa7c-5359b1203fd4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:04.952847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfc264e-f5e1-4a17-aa7c-5359b1203fd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:04.953419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfc264e-f5e1-4a17-aa7c-5359b1203fd4 admin admin] PATH_INFO: `/v3/projects/aa84af7beb3842e28c1ae71fdf935f2c/users/d3abaa919a024764bc54f0230f195874/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:04.967592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c4e451-c984-40d7-a6ed-9578be30abef admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:04.988081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c4e451-c984-40d7-a6ed-9578be30abef admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:04.999286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:05.000138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:05.000430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:05.000711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] PATH_INFO: `/v3/users/c94ec8a20ecb465a80f43a2dd27f53ef` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:05.003034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=c94ec8a20ecb465a80f43a2dd27f53ef)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:05.004881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:05.035724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfc264e-f5e1-4a17-aa7c-5359b1203fd4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa84af7beb3842e28c1ae71fdf935f2c, user_id=d3abaa919a024764bc54f0230f195874, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:05.038041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfc264e-f5e1-4a17-aa7c-5359b1203fd4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:05.063095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1150/2288] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:04 2022] PUT /identity/v3/projects/aa84af7beb3842e28c1ae71fdf935f2c/users/d3abaa919a024764bc54f0230f195874/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:05.074114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9b1ae4-6a32-4c55-a5b1-1ecc619f7676 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:05.074502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9b1ae4-6a32-4c55-a5b1-1ecc619f7676 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:05.075123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9b1ae4-6a32-4c55-a5b1-1ecc619f7676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:05.122226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c94ec8a20ecb465a80f43a2dd27f53ef'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:05.127873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c94ec8a20ecb465a80f43a2dd27f53ef'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:05.155448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c94ec8a20ecb465a80f43a2dd27f53ef'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:05.157320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c94ec8a20ecb465a80f43a2dd27f53ef'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:05.173179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b8265ab1-bdf6-4d74-9ff4-7f22ba4934ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c94ec8a20ecb465a80f43a2dd27f53ef'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:05.177770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6c9b1ae4-6a32-4c55-a5b1-1ecc619f7676 None None] MFA Rules not processed for user `d3abaa919a024764bc54f0230f195874`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:05.182325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6c9b1ae4-6a32-4c55-a5b1-1ecc619f7676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:05.221364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1139/2289] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:04 2022] DELETE /identity/v3/users/c94ec8a20ecb465a80f43a2dd27f53ef => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:56:05.228704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf843bcb-21b0-474e-b6c1-41d811f1f67e None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:05.228994 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf843bcb-21b0-474e-b6c1-41d811f1f67e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:05.229398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf843bcb-21b0-474e-b6c1-41d811f1f67e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:05.264820 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-bf843bcb-21b0-474e-b6c1-41d811f1f67e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:05.266360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-bf843bcb-21b0-474e-b6c1-41d811f1f67e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:05.356949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1151/2290] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:05 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:05.377157 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9b1ae4-6a32-4c55-a5b1-1ecc619f7676 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:05.379321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6c9b1ae4-6a32-4c55-a5b1-1ecc619f7676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:05.450334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1140/2291] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:05.466922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:05.518739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9b1ae4-6a32-4c55-a5b1-1ecc619f7676 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:05.521971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf843bcb-21b0-474e-b6c1-41d811f1f67e None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:05.524840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-bf843bcb-21b0-474e-b6c1-41d811f1f67e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:05.605807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5475bbf-b039-43f1-8a4b-842845e90157 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:05.607179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5475bbf-b039-43f1-8a4b-842845e90157 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:05.607857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5475bbf-b039-43f1-8a4b-842845e90157 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:05.609309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5475bbf-b039-43f1-8a4b-842845e90157 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:05.702351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf843bcb-21b0-474e-b6c1-41d811f1f67e None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:05.716160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee9f37a-ae70-4b11-8420-e1970aa9ab47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:05.716794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee9f37a-ae70-4b11-8420-e1970aa9ab47 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:05.722454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee9f37a-ae70-4b11-8420-e1970aa9ab47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:05.722635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee9f37a-ae70-4b11-8420-e1970aa9ab47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:05.757262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee9f37a-ae70-4b11-8420-e1970aa9ab47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:05.772668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee9f37a-ae70-4b11-8420-e1970aa9ab47 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:05.797650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1141/2292] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:56:05.834871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5475bbf-b039-43f1-8a4b-842845e90157 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:05.837323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5475bbf-b039-43f1-8a4b-842845e90157 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:05.870845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1152/2293] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:05 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:56:05.911854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee9f37a-ae70-4b11-8420-e1970aa9ab47 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:05.913707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6ee9f37a-ae70-4b11-8420-e1970aa9ab47 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:05.934401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee9f37a-ae70-4b11-8420-e1970aa9ab47 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:05.950552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:05.951207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:05.952159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:05.952159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] PATH_INFO: `/v3/users/d3abaa919a024764bc54f0230f195874` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:05.953070 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d3abaa919a024764bc54f0230f195874)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:05.955759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:05.957858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:06.024104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd3abaa919a024764bc54f0230f195874'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:06.025268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd3abaa919a024764bc54f0230f195874'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:06.037579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd3abaa919a024764bc54f0230f195874'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:06.038361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd3abaa919a024764bc54f0230f195874'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:06.052226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd3abaa919a024764bc54f0230f195874'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:06.088011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1142/2294] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:05 2022] DELETE /identity/v3/users/d3abaa919a024764bc54f0230f195874 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:06.091314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d703d061-488b-43df-900b-1ea513ed3394 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:06.091677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d703d061-488b-43df-900b-1ea513ed3394 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:06.092033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d703d061-488b-43df-900b-1ea513ed3394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:06.123944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d703d061-488b-43df-900b-1ea513ed3394 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:06.128308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d703d061-488b-43df-900b-1ea513ed3394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:06.295821 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1153/2295] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:06.341452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:06.343741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:06.492291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f622f25-708e-4574-9233-80a92c20f60f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:06.507957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd56f55c-9793-4e22-967a-08f7d2967527 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:06.508607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd56f55c-9793-4e22-967a-08f7d2967527 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:06.508889 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd56f55c-9793-4e22-967a-08f7d2967527 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:06.509161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd56f55c-9793-4e22-967a-08f7d2967527 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:06.520899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd56f55c-9793-4e22-967a-08f7d2967527 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:06.522387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd56f55c-9793-4e22-967a-08f7d2967527 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:06.538493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1143/2296] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:06.939544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d703d061-488b-43df-900b-1ea513ed3394 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:06.940556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d703d061-488b-43df-900b-1ea513ed3394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:06.957691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d703d061-488b-43df-900b-1ea513ed3394 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:06.962512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:06.963154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:06.963429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:06.963749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] PATH_INFO: `/v3/projects/548b4f8630fd47178cd4ac3dbec806ce` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:06.965214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] RBAC: Authorizing `identity:delete_project(project_id=548b4f8630fd47178cd4ac3dbec806ce)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:06.968211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:07.088904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:07.109970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '548b4f8630fd47178cd4ac3dbec806ce', 'request_id': 'req-7e8439b2-775a-4686-995c-a198da7b7655'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:07.129668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '548b4f8630fd47178cd4ac3dbec806ce', 'request_id': 'req-7e8439b2-775a-4686-995c-a198da7b7655'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:07.144725 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '548b4f8630fd47178cd4ac3dbec806ce', 'request_id': 'req-7e8439b2-775a-4686-995c-a198da7b7655'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:07.183094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] The token cache is being invalidate because project 548b4f8630fd47178cd4ac3dbec806ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:07.183313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:07.183989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:07.190389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd56f55c-9793-4e22-967a-08f7d2967527 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:07.194343 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1154/2297] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:06 2022] DELETE /identity/v3/projects/548b4f8630fd47178cd4ac3dbec806ce => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:07.232257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd56f55c-9793-4e22-967a-08f7d2967527 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:07.398366 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:07.399024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:07.399300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:07.399571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] PATH_INFO: `/v3/users/99842fc553134bf0a83f25eefcd45073` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:07.400781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=99842fc553134bf0a83f25eefcd45073)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:07.403324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:07.484256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:07.530246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99842fc553134bf0a83f25eefcd45073'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:07.531185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99842fc553134bf0a83f25eefcd45073'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:07.540866 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99842fc553134bf0a83f25eefcd45073'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:07.543713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99842fc553134bf0a83f25eefcd45073'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:07.555059 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99842fc553134bf0a83f25eefcd45073'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:07.592660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1144/2298] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:07 2022] DELETE /identity/v3/users/99842fc553134bf0a83f25eefcd45073 => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:56:07.718909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:07.723235 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:07.763807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8439b2-775a-4686-995c-a198da7b7655 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:07.941757 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:07.942502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:07.943200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:07.943200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] PATH_INFO: `/v3/projects/aa84af7beb3842e28c1ae71fdf935f2c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:07.944327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa84af7beb3842e28c1ae71fdf935f2c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:07.947927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:08.076526 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa84af7beb3842e28c1ae71fdf935f2c', 'request_id': 'req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.094749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa84af7beb3842e28c1ae71fdf935f2c', 'request_id': 'req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.107194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa84af7beb3842e28c1ae71fdf935f2c', 'request_id': 'req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.136400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] The token cache is being invalidate because project aa84af7beb3842e28c1ae71fdf935f2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:08.137229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.138121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.157000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1155/2299] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:07 2022] DELETE /identity/v3/projects/aa84af7beb3842e28c1ae71fdf935f2c => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:08.275878 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:08.277466 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:08.304022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b4f75e-e610-4b15-8e07-bcee8db497a8 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:08.408718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:08.409576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:08.409956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:08.410365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] PATH_INFO: `/v3/projects/0da0a643eafa4575a16c7820f66e848b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:08.411793 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0da0a643eafa4575a16c7820f66e848b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:08.414336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:08.468219 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:08.470693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:08.487503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe0e2e5-1b40-43d4-bc09-1ad0d982bb5b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:08.511082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0da0a643eafa4575a16c7820f66e848b', 'request_id': 'req-fde9af90-5a6a-4acf-b489-5440be273a88'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.523836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0da0a643eafa4575a16c7820f66e848b', 'request_id': 'req-fde9af90-5a6a-4acf-b489-5440be273a88'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.538680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0da0a643eafa4575a16c7820f66e848b', 'request_id': 'req-fde9af90-5a6a-4acf-b489-5440be273a88'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.553259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] The token cache is being invalidate because project 0da0a643eafa4575a16c7820f66e848b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:08.553442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.554209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-fde9af90-5a6a-4acf-b489-5440be273a88 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.556723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1145/2300] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:08 2022] DELETE /identity/v3/projects/0da0a643eafa4575a16c7820f66e848b => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:08.562475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5dcb05-e1dd-45eb-85cd-667d301b7b86 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:08.562768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5dcb05-e1dd-45eb-85cd-667d301b7b86 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:08.563237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5dcb05-e1dd-45eb-85cd-667d301b7b86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:08.578290 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:08.591295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-5f5dcb05-e1dd-45eb-85cd-667d301b7b86 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:08.598734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5f5dcb05-e1dd-45eb-85cd-667d301b7b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:08.601306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:08.602082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:08.602460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:08.602878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] PATH_INFO: `/v3/users/ee548b86b214493e83a00964137c4fde` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:08.604119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee548b86b214493e83a00964137c4fde)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:08.606373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:08.610724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1146/2301] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:08.619679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66b4f83-3c50-4a48-88a2-8f4bf700e3b1 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:08.619990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66b4f83-3c50-4a48-88a2-8f4bf700e3b1 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:08.621007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66b4f83-3c50-4a48-88a2-8f4bf700e3b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:08.660078 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f66b4f83-3c50-4a48-88a2-8f4bf700e3b1 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:08.661649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f66b4f83-3c50-4a48-88a2-8f4bf700e3b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:08.671433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee548b86b214493e83a00964137c4fde'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.680186 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1147/2302] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:08.684489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66b4f83-3c50-4a48-88a2-8f4bf700e3b1 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:08.685912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee548b86b214493e83a00964137c4fde'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.686695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee548b86b214493e83a00964137c4fde'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.687341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee548b86b214493e83a00964137c4fde'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.696612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee548b86b214493e83a00964137c4fde'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:08.697330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66b4f83-3c50-4a48-88a2-8f4bf700e3b1 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:08.705392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f0c434-a880-40f5-a057-d0c3d583ec91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:08.707143 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f0c434-a880-40f5-a057-d0c3d583ec91 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:08.707767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f0c434-a880-40f5-a057-d0c3d583ec91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:08.708112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f0c434-a880-40f5-a057-d0c3d583ec91 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:08.708847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f0c434-a880-40f5-a057-d0c3d583ec91 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:08.709370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f0c434-a880-40f5-a057-d0c3d583ec91 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:08.711239 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f0c434-a880-40f5-a057-d0c3d583ec91 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:08.726160 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1156/2303] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:08 2022] DELETE /identity/v3/users/ee548b86b214493e83a00964137c4fde => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:08.727461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1148/2304] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:56:08.731691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:08.734837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:08.842202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624824bd-ef46-4f81-b2f4-70ce5dab5d06 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:08.850481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d160fc5-61e7-4dba-adf4-eb3cb6becf41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:08.851105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d160fc5-61e7-4dba-adf4-eb3cb6becf41 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:08.851385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d160fc5-61e7-4dba-adf4-eb3cb6becf41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:08.851683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d160fc5-61e7-4dba-adf4-eb3cb6becf41 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:08.852331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d160fc5-61e7-4dba-adf4-eb3cb6becf41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:08.855424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d160fc5-61e7-4dba-adf4-eb3cb6becf41 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:08.895655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1157/2305] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:08 2022] POST /identity/v3/projects => generated 375 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:08.903920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f0c434-a880-40f5-a057-d0c3d583ec91 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:08.918944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:08.923596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f0c434-a880-40f5-a057-d0c3d583ec91 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:08.937885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d160fc5-61e7-4dba-adf4-eb3cb6becf41 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:08.940835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7d160fc5-61e7-4dba-adf4-eb3cb6becf41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:09.068671 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a66bd4-a13d-44c3-99cb-9242f580243a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:09.069416 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a66bd4-a13d-44c3-99cb-9242f580243a admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.069796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a66bd4-a13d-44c3-99cb-9242f580243a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.070200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a66bd4-a13d-44c3-99cb-9242f580243a admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.070950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a66bd4-a13d-44c3-99cb-9242f580243a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:09.072393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d160fc5-61e7-4dba-adf4-eb3cb6becf41 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:09.073061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a66bd4-a13d-44c3-99cb-9242f580243a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:09.079568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:09.081106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.081590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.082298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] PATH_INFO: `/v3/projects/3e983e7eb1474850a440d1e022c7c5a9` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.092285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e983e7eb1474850a440d1e022c7c5a9)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:09.096612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:09.105380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:09.117724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1149/2306] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:08 2022] POST /identity/v3/users => generated 288 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:09.119745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a66bd4-a13d-44c3-99cb-9242f580243a admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:09.137704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a66bd4-a13d-44c3-99cb-9242f580243a admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:09.148517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0233cd36-0292-4a0a-b116-752b64360229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:09.149207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233cd36-0292-4a0a-b116-752b64360229 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.149466 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233cd36-0292-4a0a-b116-752b64360229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.149887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0233cd36-0292-4a0a-b116-752b64360229 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.150721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0233cd36-0292-4a0a-b116-752b64360229 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:09.152372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0233cd36-0292-4a0a-b116-752b64360229 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:09.169651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1150/2307] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:09 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:09.174114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0233cd36-0292-4a0a-b116-752b64360229 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:09.185008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e983e7eb1474850a440d1e022c7c5a9', 'request_id': 'req-f0ee462a-4da0-43ff-a966-5848b4c1e484'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:09.191478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0233cd36-0292-4a0a-b116-752b64360229 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:09.203715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e983e7eb1474850a440d1e022c7c5a9', 'request_id': 'req-f0ee462a-4da0-43ff-a966-5848b4c1e484'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:09.222302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e983e7eb1474850a440d1e022c7c5a9', 'request_id': 'req-f0ee462a-4da0-43ff-a966-5848b4c1e484'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:09.244695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] The token cache is being invalidate because project 3e983e7eb1474850a440d1e022c7c5a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:09.245078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:09.246792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f0ee462a-4da0-43ff-a966-5848b4c1e484 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:09.251450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1158/2308] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:08 2022] DELETE /identity/v3/projects/3e983e7eb1474850a440d1e022c7c5a9 => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:56:09.254545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce15b1-cb64-4e7c-9734-4c2ae8a7c2a9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.254830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce15b1-cb64-4e7c-9734-4c2ae8a7c2a9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.255103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce15b1-cb64-4e7c-9734-4c2ae8a7c2a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.289562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5fce15b1-cb64-4e7c-9734-4c2ae8a7c2a9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:09.291228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5fce15b1-cb64-4e7c-9734-4c2ae8a7c2a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:09.350316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05166ed8-b61c-41ea-b5fd-42d3c9bc466d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:09.351139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05166ed8-b61c-41ea-b5fd-42d3c9bc466d admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.351485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05166ed8-b61c-41ea-b5fd-42d3c9bc466d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.351845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05166ed8-b61c-41ea-b5fd-42d3c9bc466d admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.352694 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05166ed8-b61c-41ea-b5fd-42d3c9bc466d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:09.354864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05166ed8-b61c-41ea-b5fd-42d3c9bc466d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:09.367964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1151/2309] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:09 2022] GET /identity/v3/roles => generated 1420 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:09.392558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05166ed8-b61c-41ea-b5fd-42d3c9bc466d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:09.405642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-05166ed8-b61c-41ea-b5fd-42d3c9bc466d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:09.418110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1159/2310] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:09.423999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba791083-798a-481a-bb6e-397db4691846 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.424289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba791083-798a-481a-bb6e-397db4691846 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.424711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba791083-798a-481a-bb6e-397db4691846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.435004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05166ed8-b61c-41ea-b5fd-42d3c9bc466d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:09.451344 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55ce02-99f8-4c46-a55c-5acec0090734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:09.452150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55ce02-99f8-4c46-a55c-5acec0090734 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.453477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55ce02-99f8-4c46-a55c-5acec0090734 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.453477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55ce02-99f8-4c46-a55c-5acec0090734 admin admin] PATH_INFO: `/v3/projects/c84ca0855c964555a50d3aa263410a7b/users/7613068a6ac9472a8bad6c9fc458aa70/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.457184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ba791083-798a-481a-bb6e-397db4691846 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:09.458505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ba791083-798a-481a-bb6e-397db4691846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:09.472208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1160/2311] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:09.473362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba791083-798a-481a-bb6e-397db4691846 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:09.489698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba791083-798a-481a-bb6e-397db4691846 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:09.489932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d55ce02-99f8-4c46-a55c-5acec0090734 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c84ca0855c964555a50d3aa263410a7b, user_id=7613068a6ac9472a8bad6c9fc458aa70, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:09.493811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d55ce02-99f8-4c46-a55c-5acec0090734 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:09.502939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c9468-2023-400b-8b57-79459c985c16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:09.503621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c9468-2023-400b-8b57-79459c985c16 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.503905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c9468-2023-400b-8b57-79459c985c16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.504179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c9468-2023-400b-8b57-79459c985c16 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.505055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c9468-2023-400b-8b57-79459c985c16 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:09.505381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c9468-2023-400b-8b57-79459c985c16 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:09.521245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c9468-2023-400b-8b57-79459c985c16 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:09.530101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1152/2312] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:09 2022] PUT /identity/v3/projects/c84ca0855c964555a50d3aa263410a7b/users/7613068a6ac9472a8bad6c9fc458aa70/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:09.550355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bfcb1-cc7e-4a8c-883f-b47950cf6642 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.550644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bfcb1-cc7e-4a8c-883f-b47950cf6642 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.550932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8bfcb1-cc7e-4a8c-883f-b47950cf6642 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.554443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1161/2313] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:56:09.557710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c9468-2023-400b-8b57-79459c985c16 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:09.581024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c9468-2023-400b-8b57-79459c985c16 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:09.603225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-5e8bfcb1-cc7e-4a8c-883f-b47950cf6642 None None] MFA Rules not processed for user `7613068a6ac9472a8bad6c9fc458aa70`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:09.604878 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5e8bfcb1-cc7e-4a8c-883f-b47950cf6642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:09.673759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55043b0c-0dc1-4416-aa38-7cbc5129f1c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:09.674947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55043b0c-0dc1-4416-aa38-7cbc5129f1c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.675411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55043b0c-0dc1-4416-aa38-7cbc5129f1c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.675530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55043b0c-0dc1-4416-aa38-7cbc5129f1c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.676167 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55043b0c-0dc1-4416-aa38-7cbc5129f1c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:09.678107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55043b0c-0dc1-4416-aa38-7cbc5129f1c2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:09.697859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:09.705409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:09.726308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1162/2314] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:09 2022] POST /identity/v3/projects => generated 377 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:09.727506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55043b0c-0dc1-4416-aa38-7cbc5129f1c2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:09.736888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1153/2315] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:09 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:09.742256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8bfcb1-cc7e-4a8c-883f-b47950cf6642 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:09.760719 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8bfcb1-cc7e-4a8c-883f-b47950cf6642 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:09.844569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55043b0c-0dc1-4416-aa38-7cbc5129f1c2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:09.850375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24b6936-02c1-4418-97c8-0cf842114d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:09.851334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24b6936-02c1-4418-97c8-0cf842114d4b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.851830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24b6936-02c1-4418-97c8-0cf842114d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.852655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24b6936-02c1-4418-97c8-0cf842114d4b admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.853596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24b6936-02c1-4418-97c8-0cf842114d4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:09.856034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24b6936-02c1-4418-97c8-0cf842114d4b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:09.856471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b30470-1494-4071-bc26-496982986038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:09.857172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b30470-1494-4071-bc26-496982986038 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:09.857519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b30470-1494-4071-bc26-496982986038 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:09.857858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b30470-1494-4071-bc26-496982986038 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:09.858769 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b30470-1494-4071-bc26-496982986038 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:09.863219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b30470-1494-4071-bc26-496982986038 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:09.888292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1163/2316] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:09 2022] POST /identity/v3/users => generated 289 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:09.899182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24b6936-02c1-4418-97c8-0cf842114d4b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:09.900661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1154/2317] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:09 2022] POST /identity/v3/projects => generated 379 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:09.913704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b30470-1494-4071-bc26-496982986038 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:09.921345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24b6936-02c1-4418-97c8-0cf842114d4b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.013353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b30470-1494-4071-bc26-496982986038 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.026250 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ae29b8-2f80-47e3-9ba4-d41eb89d77ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.027273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ae29b8-2f80-47e3-9ba4-d41eb89d77ef admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.027859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ae29b8-2f80-47e3-9ba4-d41eb89d77ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.028340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ae29b8-2f80-47e3-9ba4-d41eb89d77ef admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.029195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ae29b8-2f80-47e3-9ba4-d41eb89d77ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.031648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ae29b8-2f80-47e3-9ba4-d41eb89d77ef admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:10.069892 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1155/2318] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:09 2022] POST /identity/v3/users => generated 290 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:10.074621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ae29b8-2f80-47e3-9ba4-d41eb89d77ef admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:10.106391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ae29b8-2f80-47e3-9ba4-d41eb89d77ef admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.114367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b759c4-4fbb-4ca9-8e5e-a5edf2f4001d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.115349 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b759c4-4fbb-4ca9-8e5e-a5edf2f4001d admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.115803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b759c4-4fbb-4ca9-8e5e-a5edf2f4001d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.116222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b759c4-4fbb-4ca9-8e5e-a5edf2f4001d admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.117753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b759c4-4fbb-4ca9-8e5e-a5edf2f4001d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.119743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b759c4-4fbb-4ca9-8e5e-a5edf2f4001d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:10.151334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1156/2319] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:10 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:56:10.155689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b759c4-4fbb-4ca9-8e5e-a5edf2f4001d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:10.172533 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b759c4-4fbb-4ca9-8e5e-a5edf2f4001d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.179924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9471387f-1d9a-40fd-be93-3cc899935b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.180714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9471387f-1d9a-40fd-be93-3cc899935b65 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.181063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9471387f-1d9a-40fd-be93-3cc899935b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.181510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9471387f-1d9a-40fd-be93-3cc899935b65 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.182917 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9471387f-1d9a-40fd-be93-3cc899935b65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.186425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9471387f-1d9a-40fd-be93-3cc899935b65 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:10.203114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1157/2320] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:10 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:56:10.209713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9471387f-1d9a-40fd-be93-3cc899935b65 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:10.229543 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b22247-f457-4fb0-a41a-ae3ecebd53d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.232308 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9471387f-1d9a-40fd-be93-3cc899935b65 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.234766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b22247-f457-4fb0-a41a-ae3ecebd53d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.235265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b22247-f457-4fb0-a41a-ae3ecebd53d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.235732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b22247-f457-4fb0-a41a-ae3ecebd53d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.236986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0d87cf-228a-4a78-8f39-1efbe23eda33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.237828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0d87cf-228a-4a78-8f39-1efbe23eda33 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.238161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0d87cf-228a-4a78-8f39-1efbe23eda33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.238442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0d87cf-228a-4a78-8f39-1efbe23eda33 admin admin] PATH_INFO: `/v3/projects/0e5a8d0d8ecf4bcaa34199a0d99cfb1c/users/8e7adc6446f441cda2c6002beec7986e/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.238617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b22247-f457-4fb0-a41a-ae3ecebd53d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.248762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79b22247-f457-4fb0-a41a-ae3ecebd53d1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:10.261745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0d87cf-228a-4a78-8f39-1efbe23eda33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e5a8d0d8ecf4bcaa34199a0d99cfb1c, user_id=8e7adc6446f441cda2c6002beec7986e, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.264800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0d87cf-228a-4a78-8f39-1efbe23eda33 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:10.270626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1164/2321] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:09 2022] GET /identity/v3/roles => generated 1420 bytes in 372 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:10.271570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b22247-f457-4fb0-a41a-ae3ecebd53d1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:10.287920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1158/2322] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:10 2022] PUT /identity/v3/projects/0e5a8d0d8ecf4bcaa34199a0d99cfb1c/users/8e7adc6446f441cda2c6002beec7986e/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:10.290746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b22247-f457-4fb0-a41a-ae3ecebd53d1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.301209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6062b263-3ecf-4444-8b7b-5b5b75eb1576 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.301403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6062b263-3ecf-4444-8b7b-5b5b75eb1576 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.301669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6062b263-3ecf-4444-8b7b-5b5b75eb1576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.329817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:10.333599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-6062b263-3ecf-4444-8b7b-5b5b75eb1576 None None] MFA Rules not processed for user `8e7adc6446f441cda2c6002beec7986e`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:10.335822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6062b263-3ecf-4444-8b7b-5b5b75eb1576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:10.373403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1714c1fd-f256-4cf2-930f-8a9ad314a3e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.374036 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1714c1fd-f256-4cf2-930f-8a9ad314a3e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.374609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1714c1fd-f256-4cf2-930f-8a9ad314a3e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.374730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1714c1fd-f256-4cf2-930f-8a9ad314a3e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.375576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1714c1fd-f256-4cf2-930f-8a9ad314a3e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.377725 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1714c1fd-f256-4cf2-930f-8a9ad314a3e3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:10.409331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1165/2323] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:10 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:10.409331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f2133-9cbb-4fa8-8cbd-993c63b824cd None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.409331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f2133-9cbb-4fa8-8cbd-993c63b824cd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.409331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f2133-9cbb-4fa8-8cbd-993c63b824cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.426450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1159/2324] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:10 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:10.427067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6062b263-3ecf-4444-8b7b-5b5b75eb1576 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:10.441600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c29f2133-9cbb-4fa8-8cbd-993c63b824cd None None] MFA Rules not processed for user `bfee7d93eeff4e728abea067c0c42814`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:10.442903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c29f2133-9cbb-4fa8-8cbd-993c63b824cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:10.456770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6062b263-3ecf-4444-8b7b-5b5b75eb1576 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.458571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:10.462855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa84d3c-bd27-4961-9408-9ed8fe77c358 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.463993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa84d3c-bd27-4961-9408-9ed8fe77c358 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.464378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa84d3c-bd27-4961-9408-9ed8fe77c358 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.464943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa84d3c-bd27-4961-9408-9ed8fe77c358 admin admin] PATH_INFO: `/v3/projects/6f2f218a688a41ab9e06269c4be3973d/users/6415bc1076204c78ad819ab2836a3652/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.501369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa84d3c-bd27-4961-9408-9ed8fe77c358 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f2f218a688a41ab9e06269c4be3973d, user_id=6415bc1076204c78ad819ab2836a3652, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.504012 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa84d3c-bd27-4961-9408-9ed8fe77c358 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:10.521556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1160/2325] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:10 2022] PUT /identity/v3/projects/6f2f218a688a41ab9e06269c4be3973d/users/6415bc1076204c78ad819ab2836a3652/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:10.522365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa84d3c-bd27-4961-9408-9ed8fe77c358 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:10.552657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1166/2326] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:10 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:10.553971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dcd46b-1c9a-4030-8f2a-029e214788ad None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.554296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dcd46b-1c9a-4030-8f2a-029e214788ad None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.554563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96dcd46b-1c9a-4030-8f2a-029e214788ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.585393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-96dcd46b-1c9a-4030-8f2a-029e214788ad None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:10.590213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-96dcd46b-1c9a-4030-8f2a-029e214788ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:10.615818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa84d3c-bd27-4961-9408-9ed8fe77c358 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.622861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce7e49d-7748-4532-87f2-3611142f3171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.623166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7e49d-7748-4532-87f2-3611142f3171 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.623482 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7e49d-7748-4532-87f2-3611142f3171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.624108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7e49d-7748-4532-87f2-3611142f3171 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.624788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce7e49d-7748-4532-87f2-3611142f3171 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.626932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce7e49d-7748-4532-87f2-3611142f3171 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:10.680756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1161/2327] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:10 2022] POST /identity/v3/projects => generated 379 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:10.684957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce7e49d-7748-4532-87f2-3611142f3171 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:10.689013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8ce7e49d-7748-4532-87f2-3611142f3171 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:10.699859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1167/2328] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:10.700700 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dcd46b-1c9a-4030-8f2a-029e214788ad None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:10.712779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96dcd46b-1c9a-4030-8f2a-029e214788ad None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.787628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8ce600-ac6b-4391-a987-e1c5fd15bb4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.788327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8ce600-ac6b-4391-a987-e1c5fd15bb4f admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.788839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8ce600-ac6b-4391-a987-e1c5fd15bb4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.789117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f8ce600-ac6b-4391-a987-e1c5fd15bb4f admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.789393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce7e49d-7748-4532-87f2-3611142f3171 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.789812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8ce600-ac6b-4391-a987-e1c5fd15bb4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.791888 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f8ce600-ac6b-4391-a987-e1c5fd15bb4f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:10.795228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb3ea87-8242-483d-8943-b1f06dc7b96f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.795826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb3ea87-8242-483d-8943-b1f06dc7b96f admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.796150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb3ea87-8242-483d-8943-b1f06dc7b96f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.796615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb3ea87-8242-483d-8943-b1f06dc7b96f admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.797280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb3ea87-8242-483d-8943-b1f06dc7b96f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.806859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb3ea87-8242-483d-8943-b1f06dc7b96f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:10.837799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:10.854640 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1168/2329] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:10 2022] POST /identity/v3/users => generated 296 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:10.861611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8ce600-ac6b-4391-a987-e1c5fd15bb4f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:10.864073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6f8ce600-ac6b-4391-a987-e1c5fd15bb4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:10.896650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1162/2330] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:10 2022] POST /identity/v3/projects => generated 371 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:10.896650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb3ea87-8242-483d-8943-b1f06dc7b96f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:10.949277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:10.968627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f8ce600-ac6b-4391-a987-e1c5fd15bb4f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.988572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb3ea87-8242-483d-8943-b1f06dc7b96f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:10.994336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b97d8fd-48df-4bbc-a75a-2fc5a16957a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:10.995166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b97d8fd-48df-4bbc-a75a-2fc5a16957a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:10.995818 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b97d8fd-48df-4bbc-a75a-2fc5a16957a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:10.996441 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b97d8fd-48df-4bbc-a75a-2fc5a16957a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:10.997120 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b97d8fd-48df-4bbc-a75a-2fc5a16957a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:10.999146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b97d8fd-48df-4bbc-a75a-2fc5a16957a3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:11.025068 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1163/2331] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:10 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:11.025930 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b97d8fd-48df-4bbc-a75a-2fc5a16957a3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:11.035446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf8f0c7-3fa5-454b-bc74-efb42e7fced7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:11.036494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf8f0c7-3fa5-454b-bc74-efb42e7fced7 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.036494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf8f0c7-3fa5-454b-bc74-efb42e7fced7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.037065 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf8f0c7-3fa5-454b-bc74-efb42e7fced7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.043476 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b97d8fd-48df-4bbc-a75a-2fc5a16957a3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:11.049567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e8057-7a81-4180-ac65-6a445a7919bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:11.050274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e8057-7a81-4180-ac65-6a445a7919bc admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.050611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e8057-7a81-4180-ac65-6a445a7919bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.050878 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e8057-7a81-4180-ac65-6a445a7919bc admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.051611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241e8057-7a81-4180-ac65-6a445a7919bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:11.055268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241e8057-7a81-4180-ac65-6a445a7919bc admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:11.055463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf8f0c7-3fa5-454b-bc74-efb42e7fced7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:11.057056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf8f0c7-3fa5-454b-bc74-efb42e7fced7 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:11.078930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1169/2332] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:11.080179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf8f0c7-3fa5-454b-bc74-efb42e7fced7 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:11.081734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1164/2333] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:11 2022] POST /identity/v3/users => generated 292 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:11.087943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e8057-7a81-4180-ac65-6a445a7919bc admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:11.092008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf8f0c7-3fa5-454b-bc74-efb42e7fced7 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:11.097838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55b964c-e179-4eb3-b9dc-192aa95c9d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:11.098540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55b964c-e179-4eb3-b9dc-192aa95c9d2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.098911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55b964c-e179-4eb3-b9dc-192aa95c9d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.099250 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55b964c-e179-4eb3-b9dc-192aa95c9d2e admin admin] PATH_INFO: `/v3/projects/9074c02e64b44fabb15efb838e2f4c75/users/dbe2ca0560f24020a2145a6e9958b854/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.101946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e8057-7a81-4180-ac65-6a445a7919bc admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:11.110124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe48a69-1d10-4c5e-976f-cb7d54901aa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:11.111647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe48a69-1d10-4c5e-976f-cb7d54901aa9 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.112073 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe48a69-1d10-4c5e-976f-cb7d54901aa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.112486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe48a69-1d10-4c5e-976f-cb7d54901aa9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.113584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe48a69-1d10-4c5e-976f-cb7d54901aa9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:11.115551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe48a69-1d10-4c5e-976f-cb7d54901aa9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:11.117376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55b964c-e179-4eb3-b9dc-192aa95c9d2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9074c02e64b44fabb15efb838e2f4c75, user_id=dbe2ca0560f24020a2145a6e9958b854, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:11.120759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55b964c-e179-4eb3-b9dc-192aa95c9d2e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:11.132374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1165/2334] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:11 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:11.137067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe48a69-1d10-4c5e-976f-cb7d54901aa9 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:11.138910 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1170/2335] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:11 2022] PUT /identity/v3/projects/9074c02e64b44fabb15efb838e2f4c75/users/dbe2ca0560f24020a2145a6e9958b854/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:11.144219 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55b964c-e179-4eb3-b9dc-192aa95c9d2e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:11.165959 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe48a69-1d10-4c5e-976f-cb7d54901aa9 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:11.263691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55b964c-e179-4eb3-b9dc-192aa95c9d2e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:11.271540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0223d5eb-1606-4c82-adb4-afa32c9929de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:11.272503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0223d5eb-1606-4c82-adb4-afa32c9929de admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.273577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0223d5eb-1606-4c82-adb4-afa32c9929de admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.273577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0223d5eb-1606-4c82-adb4-afa32c9929de admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.273798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0223d5eb-1606-4c82-adb4-afa32c9929de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:11.275401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0223d5eb-1606-4c82-adb4-afa32c9929de admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:11.275515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcacbd5-7958-4a02-8c34-1285d9435c8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:11.276237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcacbd5-7958-4a02-8c34-1285d9435c8b admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.276522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcacbd5-7958-4a02-8c34-1285d9435c8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.277077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcacbd5-7958-4a02-8c34-1285d9435c8b admin admin] PATH_INFO: `/v3/projects/0e6bb29020b04fc2b21ca50a53e3b59f/users/b0f9f74f9f3b48e894bd6c3aea35aa3b/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.287720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1171/2336] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:11 2022] GET /identity/v3/roles => generated 1420 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:11.292583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0223d5eb-1606-4c82-adb4-afa32c9929de admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:11.301259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcacbd5-7958-4a02-8c34-1285d9435c8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e6bb29020b04fc2b21ca50a53e3b59f, user_id=b0f9f74f9f3b48e894bd6c3aea35aa3b, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:11.302934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcacbd5-7958-4a02-8c34-1285d9435c8b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:11.312490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0223d5eb-1606-4c82-adb4-afa32c9929de admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:11.321328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995359f2-573a-4ad7-994b-b29e944ea87a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:11.326667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995359f2-573a-4ad7-994b-b29e944ea87a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.327151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995359f2-573a-4ad7-994b-b29e944ea87a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.327531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995359f2-573a-4ad7-994b-b29e944ea87a admin admin] PATH_INFO: `/v3/projects/9074c02e64b44fabb15efb838e2f4c75/users/dbe2ca0560f24020a2145a6e9958b854/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.330914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995359f2-573a-4ad7-994b-b29e944ea87a admin admin] RBAC: Authorizing `identity:create_grant(project_id=9074c02e64b44fabb15efb838e2f4c75, user_id=dbe2ca0560f24020a2145a6e9958b854, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:11.335495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1166/2337] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:11 2022] PUT /identity/v3/projects/0e6bb29020b04fc2b21ca50a53e3b59f/users/b0f9f74f9f3b48e894bd6c3aea35aa3b/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:11.336492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995359f2-573a-4ad7-994b-b29e944ea87a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:11.336794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcacbd5-7958-4a02-8c34-1285d9435c8b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:11.356138 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1172/2338] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:11 2022] PUT /identity/v3/projects/9074c02e64b44fabb15efb838e2f4c75/users/dbe2ca0560f24020a2145a6e9958b854/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:11.366936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7359e006-e856-464c-9acf-3393446a7ef8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.367316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7359e006-e856-464c-9acf-3393446a7ef8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.367593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7359e006-e856-464c-9acf-3393446a7ef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.402631 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:11.407215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7359e006-e856-464c-9acf-3393446a7ef8 None None] MFA Rules not processed for user `dbe2ca0560f24020a2145a6e9958b854`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:11.409400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7359e006-e856-464c-9acf-3393446a7ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:11.437196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcacbd5-7958-4a02-8c34-1285d9435c8b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:11.484889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:11.566367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a577a9bb-5235-4f5c-bf9c-5a6c70f370fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:11.567409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a577a9bb-5235-4f5c-bf9c-5a6c70f370fb admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.567409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a577a9bb-5235-4f5c-bf9c-5a6c70f370fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.569643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a577a9bb-5235-4f5c-bf9c-5a6c70f370fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.573581 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a577a9bb-5235-4f5c-bf9c-5a6c70f370fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:11.578266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a577a9bb-5235-4f5c-bf9c-5a6c70f370fb admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:11.592703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1173/2339] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:11 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:11.598110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1167/2340] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:11 2022] GET /identity/v3/roles => generated 1420 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:56:11.601815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7359e006-e856-464c-9acf-3393446a7ef8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:11.603080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a577a9bb-5235-4f5c-bf9c-5a6c70f370fb admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:11.603520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7359e006-e856-464c-9acf-3393446a7ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:11.624228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a577a9bb-5235-4f5c-bf9c-5a6c70f370fb admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:11.632836 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f62ed0-c495-40df-ad6f-34f208211c5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:11.649872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f62ed0-c495-40df-ad6f-34f208211c5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.650728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f62ed0-c495-40df-ad6f-34f208211c5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.651164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f62ed0-c495-40df-ad6f-34f208211c5a admin admin] PATH_INFO: `/v3/projects/0e6bb29020b04fc2b21ca50a53e3b59f/users/b0f9f74f9f3b48e894bd6c3aea35aa3b/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.654902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f62ed0-c495-40df-ad6f-34f208211c5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e6bb29020b04fc2b21ca50a53e3b59f, user_id=b0f9f74f9f3b48e894bd6c3aea35aa3b, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:11.659689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f62ed0-c495-40df-ad6f-34f208211c5a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:11.675680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1168/2341] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:11 2022] PUT /identity/v3/projects/0e6bb29020b04fc2b21ca50a53e3b59f/users/b0f9f74f9f3b48e894bd6c3aea35aa3b/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:11.692158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7359e006-e856-464c-9acf-3393446a7ef8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:11.797803 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42635df8-aa05-4c7c-b049-2afeeecae932 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:11.798552 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42635df8-aa05-4c7c-b049-2afeeecae932 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:11.798840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42635df8-aa05-4c7c-b049-2afeeecae932 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:11.799157 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42635df8-aa05-4c7c-b049-2afeeecae932 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:11.959171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42635df8-aa05-4c7c-b049-2afeeecae932 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:11.961462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42635df8-aa05-4c7c-b049-2afeeecae932 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:12.025410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1174/2342] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:11 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:12.132470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f62ed0-c495-40df-ad6f-34f208211c5a admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:12.153478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f62ed0-c495-40df-ad6f-34f208211c5a admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:12.170007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830fe5f9-24cb-4dda-a391-fae0481b1a5f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:12.171784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830fe5f9-24cb-4dda-a391-fae0481b1a5f service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:12.173142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830fe5f9-24cb-4dda-a391-fae0481b1a5f service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:12.174259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830fe5f9-24cb-4dda-a391-fae0481b1a5f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:12.293255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830fe5f9-24cb-4dda-a391-fae0481b1a5f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:12.297056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830fe5f9-24cb-4dda-a391-fae0481b1a5f service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:12.328797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1169/2343] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:12 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:18.559673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42635df8-aa05-4c7c-b049-2afeeecae932 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:18.588264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc369571-3efa-4a17-94db-04f7a096f4e0 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:18.589598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc369571-3efa-4a17-94db-04f7a096f4e0 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:18.590216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc369571-3efa-4a17-94db-04f7a096f4e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:18.641183 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-fc369571-3efa-4a17-94db-04f7a096f4e0 None None] MFA Rules not processed for user `6415bc1076204c78ad819ab2836a3652`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:18.644620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fc369571-3efa-4a17-94db-04f7a096f4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:18.680902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:18.709006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42635df8-aa05-4c7c-b049-2afeeecae932 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:18.723638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:18.724473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:18.725332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:18.725332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] PATH_INFO: `/v3/users/7613068a6ac9472a8bad6c9fc458aa70` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:18.727397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] RBAC: Authorizing `identity:delete_user(user_id=7613068a6ac9472a8bad6c9fc458aa70)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:18.732766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:18.783494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:18.804298 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1170/2344] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:18 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:18.819037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc369571-3efa-4a17-94db-04f7a096f4e0 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:18.820533 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fc369571-3efa-4a17-94db-04f7a096f4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:18.826020 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7613068a6ac9472a8bad6c9fc458aa70'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:18.835867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7613068a6ac9472a8bad6c9fc458aa70'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:18.843355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7613068a6ac9472a8bad6c9fc458aa70'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:18.845015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7613068a6ac9472a8bad6c9fc458aa70'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:18.878632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-b69486e6-08e3-4872-99d9-0aebdb81614b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7613068a6ac9472a8bad6c9fc458aa70'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:18.960889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1175/2345] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:18 2022] DELETE /identity/v3/users/7613068a6ac9472a8bad6c9fc458aa70 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:18.968384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b083ef8-f816-4be3-b4bf-a58dcad90760 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:18.968771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b083ef8-f816-4be3-b4bf-a58dcad90760 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:18.969156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b083ef8-f816-4be3-b4bf-a58dcad90760 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:19.093304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9b083ef8-f816-4be3-b4bf-a58dcad90760 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:19.103600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9b083ef8-f816-4be3-b4bf-a58dcad90760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:19.113121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc369571-3efa-4a17-94db-04f7a096f4e0 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:19.303536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1176/2346] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:19.304880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a17b40-7b26-4e14-b6db-f8a86020b0a8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:19.305258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a17b40-7b26-4e14-b6db-f8a86020b0a8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:19.305432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a17b40-7b26-4e14-b6db-f8a86020b0a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:19.381567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1ed9c4-d7f7-40cf-a37f-f0a559ae342d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:19.382275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1ed9c4-d7f7-40cf-a37f-f0a559ae342d service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:19.382546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1ed9c4-d7f7-40cf-a37f-f0a559ae342d service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:19.382887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1ed9c4-d7f7-40cf-a37f-f0a559ae342d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:19.386881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fe1ed9c4-d7f7-40cf-a37f-f0a559ae342d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:19.400699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f4a17b40-7b26-4e14-b6db-f8a86020b0a8 None None] MFA Rules not processed for user `b0f9f74f9f3b48e894bd6c3aea35aa3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:19.406170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f4a17b40-7b26-4e14-b6db-f8a86020b0a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:19.544727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1ed9c4-d7f7-40cf-a37f-f0a559ae342d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:19.546591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe1ed9c4-d7f7-40cf-a37f-f0a559ae342d service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:19.557161 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:19.566783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1171/2347] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:18 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 748 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:19.568849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1ed9c4-d7f7-40cf-a37f-f0a559ae342d service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:19.574198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fe1ed9c4-d7f7-40cf-a37f-f0a559ae342d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:19.597316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1177/2348] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:19 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:19.685698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a17b40-7b26-4e14-b6db-f8a86020b0a8 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:19.687175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f4a17b40-7b26-4e14-b6db-f8a86020b0a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:19.697398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:19.741342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1ed9c4-d7f7-40cf-a37f-f0a559ae342d service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:19.742703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a17b40-7b26-4e14-b6db-f8a86020b0a8 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:19.747336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a270490c-137a-4624-8301-6d589bab8c36 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:19.747712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a270490c-137a-4624-8301-6d589bab8c36 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:19.748029 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a270490c-137a-4624-8301-6d589bab8c36 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:19.748452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a270490c-137a-4624-8301-6d589bab8c36 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:19.762092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606cbfd7-91c7-461a-a662-200ebda5305c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:19.764793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606cbfd7-91c7-461a-a662-200ebda5305c service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:19.765077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606cbfd7-91c7-461a-a662-200ebda5305c service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:19.765356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606cbfd7-91c7-461a-a662-200ebda5305c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:19.768843 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a270490c-137a-4624-8301-6d589bab8c36 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:19.770337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a270490c-137a-4624-8301-6d589bab8c36 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:19.772852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-606cbfd7-91c7-461a-a662-200ebda5305c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:19.789531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1172/2349] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:19.940208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606cbfd7-91c7-461a-a662-200ebda5305c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:19.951439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606cbfd7-91c7-461a-a662-200ebda5305c service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:19.966142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1178/2350] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:19 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:21.009316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a270490c-137a-4624-8301-6d589bab8c36 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:21.011071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a270490c-137a-4624-8301-6d589bab8c36 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:21.047320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a270490c-137a-4624-8301-6d589bab8c36 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:21.056611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:21.057357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:21.057720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:21.058287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] PATH_INFO: `/v3/projects/c84ca0855c964555a50d3aa263410a7b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:21.061508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=c84ca0855c964555a50d3aa263410a7b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:21.066862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:21.166659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c84ca0855c964555a50d3aa263410a7b', 'request_id': 'req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:21.182657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c84ca0855c964555a50d3aa263410a7b', 'request_id': 'req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:21.195110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c84ca0855c964555a50d3aa263410a7b', 'request_id': 'req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:21.216856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] The token cache is being invalidate because project c84ca0855c964555a50d3aa263410a7b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:21.217649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:21.219122 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:21.222902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1173/2351] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:21 2022] DELETE /identity/v3/projects/c84ca0855c964555a50d3aa263410a7b => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:21.226836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606cbfd7-91c7-461a-a662-200ebda5305c service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:21.239433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606cbfd7-91c7-461a-a662-200ebda5305c service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:21.347624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:21.348288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:21.348563 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:21.348835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] PATH_INFO: `/v3/users/8e7adc6446f441cda2c6002beec7986e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:21.351225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e7adc6446f441cda2c6002beec7986e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:21.354186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:21.450381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e7adc6446f441cda2c6002beec7986e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:21.460397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e7adc6446f441cda2c6002beec7986e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:21.461842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e7adc6446f441cda2c6002beec7986e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:21.462896 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e7adc6446f441cda2c6002beec7986e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:21.473382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-cdc326a0-409c-4999-b4c0-ca456395643e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e7adc6446f441cda2c6002beec7986e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:21.496002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1179/2352] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:21 2022] DELETE /identity/v3/users/8e7adc6446f441cda2c6002beec7986e => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:22.016447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:22.019171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:22.059868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1e3529-f204-4f7b-ab7c-0f009abad5ce admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:22.136199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ef371b-92a9-4742-81c1-54a2286ba4e5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:22.136483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ef371b-92a9-4742-81c1-54a2286ba4e5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:22.136759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ef371b-92a9-4742-81c1-54a2286ba4e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:22.160770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-42ef371b-92a9-4742-81c1-54a2286ba4e5 None None] MFA Rules not processed for user `bfee7d93eeff4e728abea067c0c42814`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:22.162252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-42ef371b-92a9-4742-81c1-54a2286ba4e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:22.168659 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:22.183705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:22.184634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:22.185072 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:22.185523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] PATH_INFO: `/v3/projects/0e5a8d0d8ecf4bcaa34199a0d99cfb1c` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:22.187113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e5a8d0d8ecf4bcaa34199a0d99cfb1c)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:22.190147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:22.237874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1180/2353] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:22 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:22.260587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e5a8d0d8ecf4bcaa34199a0d99cfb1c', 'request_id': 'req-26c0d5b3-dd10-4844-b3af-1e61b075d648'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:22.276325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e5a8d0d8ecf4bcaa34199a0d99cfb1c', 'request_id': 'req-26c0d5b3-dd10-4844-b3af-1e61b075d648'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:22.291954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e5a8d0d8ecf4bcaa34199a0d99cfb1c', 'request_id': 'req-26c0d5b3-dd10-4844-b3af-1e61b075d648'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:22.295364 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:22.314977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] The token cache is being invalidate because project 0e5a8d0d8ecf4bcaa34199a0d99cfb1c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:22.315519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:22.316949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26c0d5b3-dd10-4844-b3af-1e61b075d648 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:22.319134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1174/2354] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:22 2022] DELETE /identity/v3/projects/0e5a8d0d8ecf4bcaa34199a0d99cfb1c => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:22.322962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ef371b-92a9-4742-81c1-54a2286ba4e5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:22.324521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-42ef371b-92a9-4742-81c1-54a2286ba4e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:22.365627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ef371b-92a9-4742-81c1-54a2286ba4e5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:22.491291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:22.492680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:22.493125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:22.493573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] PATH_INFO: `/v3/users/dbe2ca0560f24020a2145a6e9958b854` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:22.495667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dbe2ca0560f24020a2145a6e9958b854)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:22.497847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:22.611075 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dbe2ca0560f24020a2145a6e9958b854'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:22.627112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dbe2ca0560f24020a2145a6e9958b854'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:22.628049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dbe2ca0560f24020a2145a6e9958b854'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:22.628806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dbe2ca0560f24020a2145a6e9958b854'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:22.656811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dbe2ca0560f24020a2145a6e9958b854'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:22.696003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1181/2355] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:22 2022] DELETE /identity/v3/users/dbe2ca0560f24020a2145a6e9958b854 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:22.756253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8828e443-0964-49ca-b5a8-376ebb15d660 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:22.757137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8828e443-0964-49ca-b5a8-376ebb15d660 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:22.757507 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8828e443-0964-49ca-b5a8-376ebb15d660 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:22.786985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8828e443-0964-49ca-b5a8-376ebb15d660 None None] MFA Rules not processed for user `6415bc1076204c78ad819ab2836a3652`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:22.788780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8828e443-0964-49ca-b5a8-376ebb15d660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:22.878005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1175/2356] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:23.364053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:23.366166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:23.434901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:23.480768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f5af76-ce0c-4caf-ac4c-fe11f9e295d0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:23.492987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:23.493875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:23.494346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:23.494759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] PATH_INFO: `/v3/users/6ee08a1127ee40f78e6a6237da0f5ce8` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:23.498804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6ee08a1127ee40f78e6a6237da0f5ce8)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:23.502844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:23.592832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6ee08a1127ee40f78e6a6237da0f5ce8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:23.603577 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6ee08a1127ee40f78e6a6237da0f5ce8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:23.604092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6ee08a1127ee40f78e6a6237da0f5ce8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:23.604701 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6ee08a1127ee40f78e6a6237da0f5ce8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:23.616909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ed128e45-6074-4f30-a5a9-5a6dede6bf27 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6ee08a1127ee40f78e6a6237da0f5ce8'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:23.651990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1182/2357] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:23 2022] DELETE /identity/v3/users/6ee08a1127ee40f78e6a6237da0f5ce8 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:56:23.652367 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8828e443-0964-49ca-b5a8-376ebb15d660 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:23.654042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8828e443-0964-49ca-b5a8-376ebb15d660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:23.657841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fda245-4ff0-4e81-836d-9832f4abd6f1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:23.658146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fda245-4ff0-4e81-836d-9832f4abd6f1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:23.658416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fda245-4ff0-4e81-836d-9832f4abd6f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:23.690209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f6fda245-4ff0-4e81-836d-9832f4abd6f1 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:23.691539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f6fda245-4ff0-4e81-836d-9832f4abd6f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:23.696408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8828e443-0964-49ca-b5a8-376ebb15d660 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:23.779358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1183/2358] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:23.791507 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:23.792539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:23.792819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:23.793095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] PATH_INFO: `/v3/projects/9074c02e64b44fabb15efb838e2f4c75` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:23.794520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9074c02e64b44fabb15efb838e2f4c75)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:23.796625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:23.796761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fda245-4ff0-4e81-836d-9832f4abd6f1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:23.803280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f6fda245-4ff0-4e81-836d-9832f4abd6f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:23.856050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9074c02e64b44fabb15efb838e2f4c75', 'request_id': 'req-4d22c0c9-f159-4bb2-9077-cfc141572b56'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:23.864902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9074c02e64b44fabb15efb838e2f4c75', 'request_id': 'req-4d22c0c9-f159-4bb2-9077-cfc141572b56'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:23.871952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9074c02e64b44fabb15efb838e2f4c75', 'request_id': 'req-4d22c0c9-f159-4bb2-9077-cfc141572b56'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:23.883133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] The token cache is being invalidate because project 9074c02e64b44fabb15efb838e2f4c75 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:23.884045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:23.884045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4d22c0c9-f159-4bb2-9077-cfc141572b56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:23.885794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1176/2359] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:23 2022] DELETE /identity/v3/projects/9074c02e64b44fabb15efb838e2f4c75 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:56:23.893489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fda245-4ff0-4e81-836d-9832f4abd6f1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:23.894098 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9229c01f-da27-4c8b-a899-15b4c4da9511 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:23.894478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9229c01f-da27-4c8b-a899-15b4c4da9511 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:23.894762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9229c01f-da27-4c8b-a899-15b4c4da9511 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:23.905069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:23.918307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9229c01f-da27-4c8b-a899-15b4c4da9511 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:23.919649 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9229c01f-da27-4c8b-a899-15b4c4da9511 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:23.948337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:23.962704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33efddd-addb-444d-be03-94170a0836ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:23.963380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33efddd-addb-444d-be03-94170a0836ef service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:23.963662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33efddd-addb-444d-be03-94170a0836ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:23.963939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33efddd-addb-444d-be03-94170a0836ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:23.965757 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e33efddd-addb-444d-be03-94170a0836ef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:24.050998 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1177/2360] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:24.059486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe7d937-20ce-47a6-80ff-f1618622a964 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.059857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe7d937-20ce-47a6-80ff-f1618622a964 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.060209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe7d937-20ce-47a6-80ff-f1618622a964 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.092462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33efddd-addb-444d-be03-94170a0836ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:24.099588 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e33efddd-addb-444d-be03-94170a0836ef service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:24.105278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-cbe7d937-20ce-47a6-80ff-f1618622a964 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:24.107174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cbe7d937-20ce-47a6-80ff-f1618622a964 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:24.113289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1184/2361] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:56:24.117330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1178/2362] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:24.122177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33efddd-addb-444d-be03-94170a0836ef service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:24.139211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33efddd-addb-444d-be03-94170a0836ef service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:24.146302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a19737f-2aab-47f1-9713-7f7f7a4c4237 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:24.147022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a19737f-2aab-47f1-9713-7f7f7a4c4237 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.147384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a19737f-2aab-47f1-9713-7f7f7a4c4237 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.147656 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a19737f-2aab-47f1-9713-7f7f7a4c4237 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.148332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a19737f-2aab-47f1-9713-7f7f7a4c4237 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:24.148635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a19737f-2aab-47f1-9713-7f7f7a4c4237 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:24.150658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a19737f-2aab-47f1-9713-7f7f7a4c4237 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:24.167148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1185/2363] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:56:24.169028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe7d937-20ce-47a6-80ff-f1618622a964 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:24.185249 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe7d937-20ce-47a6-80ff-f1618622a964 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:24.197034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11046d2-ee78-4407-ae5c-376108b61206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:24.197666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11046d2-ee78-4407-ae5c-376108b61206 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.198010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11046d2-ee78-4407-ae5c-376108b61206 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.198280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11046d2-ee78-4407-ae5c-376108b61206 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.199063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11046d2-ee78-4407-ae5c-376108b61206 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:24.200965 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11046d2-ee78-4407-ae5c-376108b61206 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:24.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1179/2364] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:24 2022] POST /identity/v3/projects => generated 373 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:24.260051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a19737f-2aab-47f1-9713-7f7f7a4c4237 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:24.277456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a19737f-2aab-47f1-9713-7f7f7a4c4237 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:24.386591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd1accb-1cdf-4057-912f-31c835361ce5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:24.387241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd1accb-1cdf-4057-912f-31c835361ce5 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.387463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd1accb-1cdf-4057-912f-31c835361ce5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.387736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd1accb-1cdf-4057-912f-31c835361ce5 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.388363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd1accb-1cdf-4057-912f-31c835361ce5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:24.390321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd1accb-1cdf-4057-912f-31c835361ce5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:24.434443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1186/2365] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:24 2022] POST /identity/v3/users => generated 287 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:24.436371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11046d2-ee78-4407-ae5c-376108b61206 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:24.464808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11046d2-ee78-4407-ae5c-376108b61206 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:24.472823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd1accb-1cdf-4057-912f-31c835361ce5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:24.478045 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-acd1accb-1cdf-4057-912f-31c835361ce5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:24.483071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d31d32b-6b45-42ff-b304-3b24d9eb6698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:24.483562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31d32b-6b45-42ff-b304-3b24d9eb6698 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.483837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31d32b-6b45-42ff-b304-3b24d9eb6698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.484106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31d32b-6b45-42ff-b304-3b24d9eb6698 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.486499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d31d32b-6b45-42ff-b304-3b24d9eb6698 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:24.489924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d31d32b-6b45-42ff-b304-3b24d9eb6698 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:24.515814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1180/2366] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:24.517522 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d31d32b-6b45-42ff-b304-3b24d9eb6698 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:24.534091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd1accb-1cdf-4057-912f-31c835361ce5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:24.542809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:24.543541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.543817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.544095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] PATH_INFO: `/v3/projects/dd7f11d7fea44639a879801793f1165b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.550937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd7f11d7fea44639a879801793f1165b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:24.553070 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:24.553194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d31d32b-6b45-42ff-b304-3b24d9eb6698 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:24.567011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1a86e6-7974-4a79-ada4-0de59a4a063a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:24.568129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1a86e6-7974-4a79-ada4-0de59a4a063a admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.570746 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1a86e6-7974-4a79-ada4-0de59a4a063a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.571154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1a86e6-7974-4a79-ada4-0de59a4a063a admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.572080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1a86e6-7974-4a79-ada4-0de59a4a063a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:24.578157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1a86e6-7974-4a79-ada4-0de59a4a063a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:24.618884 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1181/2367] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:24 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:24.624680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=3,keystone:DELETE=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:24.625723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1a86e6-7974-4a79-ada4-0de59a4a063a admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:24.637635 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1a86e6-7974-4a79-ada4-0de59a4a063a admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:24.673366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd7f11d7fea44639a879801793f1165b', 'request_id': 'req-9550e1a0-1012-4e3e-96a7-c01253e07f56'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:24.685452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd7f11d7fea44639a879801793f1165b', 'request_id': 'req-9550e1a0-1012-4e3e-96a7-c01253e07f56'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:24.708281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd7f11d7fea44639a879801793f1165b', 'request_id': 'req-9550e1a0-1012-4e3e-96a7-c01253e07f56'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:24.740426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] The token cache is being invalidate because project dd7f11d7fea44639a879801793f1165b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:24.740796 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:24.744887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-9550e1a0-1012-4e3e-96a7-c01253e07f56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:24.747849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1187/2368] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:24 2022] DELETE /identity/v3/projects/dd7f11d7fea44639a879801793f1165b => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:24.754916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9bfb54-27bd-4f2f-9f7f-900f4ed4156a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.755209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9bfb54-27bd-4f2f-9f7f-900f4ed4156a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.755668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9bfb54-27bd-4f2f-9f7f-900f4ed4156a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.771044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b28a6a9-18d9-4674-b142-4ff65d0c2f92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:24.772315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b28a6a9-18d9-4674-b142-4ff65d0c2f92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.772606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b28a6a9-18d9-4674-b142-4ff65d0c2f92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.772890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b28a6a9-18d9-4674-b142-4ff65d0c2f92 admin admin] PATH_INFO: `/v3/projects/e8b83d2c37b747b980257937fba802bd/users/6b0d44b356f84a75a1e49de5ecfef8fe/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.778385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6a9bfb54-27bd-4f2f-9f7f-900f4ed4156a None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:24.793865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b28a6a9-18d9-4674-b142-4ff65d0c2f92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8b83d2c37b747b980257937fba802bd, user_id=6b0d44b356f84a75a1e49de5ecfef8fe, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:24.794363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6a9bfb54-27bd-4f2f-9f7f-900f4ed4156a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:24.800454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b28a6a9-18d9-4674-b142-4ff65d0c2f92 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:24.812204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1188/2369] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:24.820012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c42a3f-a023-490c-9738-3ecf4ad52552 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.820416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c42a3f-a023-490c-9738-3ecf4ad52552 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.820694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c42a3f-a023-490c-9738-3ecf4ad52552 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.843431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1182/2370] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:24 2022] PUT /identity/v3/projects/e8b83d2c37b747b980257937fba802bd/users/6b0d44b356f84a75a1e49de5ecfef8fe/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:24.848232 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb184c0-a114-431c-8a36-28d3d2fa9db7 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.848510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb184c0-a114-431c-8a36-28d3d2fa9db7 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.849040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb184c0-a114-431c-8a36-28d3d2fa9db7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.851954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-61c42a3f-a023-490c-9738-3ecf4ad52552 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:24.853576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-61c42a3f-a023-490c-9738-3ecf4ad52552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:24.894165 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-fcb184c0-a114-431c-8a36-28d3d2fa9db7 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:24.896387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fcb184c0-a114-431c-8a36-28d3d2fa9db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:24.936096 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1189/2371] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:24.943019 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c42a3f-a023-490c-9738-3ecf4ad52552 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:24.966128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c42a3f-a023-490c-9738-3ecf4ad52552 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:24.972569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572cc898-59ee-4351-91c7-156351a54672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:24.973385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572cc898-59ee-4351-91c7-156351a54672 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:24.973664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572cc898-59ee-4351-91c7-156351a54672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:24.974201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572cc898-59ee-4351-91c7-156351a54672 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:24.974858 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572cc898-59ee-4351-91c7-156351a54672 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:24.975190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572cc898-59ee-4351-91c7-156351a54672 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:24.977121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572cc898-59ee-4351-91c7-156351a54672 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:24.981923 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:24.993142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1190/2372] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:56:24.998592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572cc898-59ee-4351-91c7-156351a54672 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:25.008085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1183/2373] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:25.015607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb184c0-a114-431c-8a36-28d3d2fa9db7 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:25.018636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fcb184c0-a114-431c-8a36-28d3d2fa9db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:25.020779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572cc898-59ee-4351-91c7-156351a54672 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:25.026452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6a039a-2975-4c90-92d2-a5e3136007b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:25.027139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6a039a-2975-4c90-92d2-a5e3136007b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:25.027482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6a039a-2975-4c90-92d2-a5e3136007b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:25.027949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6a039a-2975-4c90-92d2-a5e3136007b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:25.028848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6a039a-2975-4c90-92d2-a5e3136007b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:25.030927 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6a039a-2975-4c90-92d2-a5e3136007b3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:25.083510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1191/2374] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:24 2022] POST /identity/v3/projects => generated 371 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:56:25.090545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6a039a-2975-4c90-92d2-a5e3136007b3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:25.128001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb184c0-a114-431c-8a36-28d3d2fa9db7 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:25.219478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6a039a-2975-4c90-92d2-a5e3136007b3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:25.221728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55866cc7-fc03-4dea-9f03-6a65cae39192 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:25.222552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55866cc7-fc03-4dea-9f03-6a65cae39192 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:25.222847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55866cc7-fc03-4dea-9f03-6a65cae39192 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:25.223106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55866cc7-fc03-4dea-9f03-6a65cae39192 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:25.230406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039313db-d382-4cc6-9f1e-34157bcee6de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:25.231404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039313db-d382-4cc6-9f1e-34157bcee6de admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:25.232716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039313db-d382-4cc6-9f1e-34157bcee6de admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:25.233190 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039313db-d382-4cc6-9f1e-34157bcee6de admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:25.234231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039313db-d382-4cc6-9f1e-34157bcee6de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:25.237077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039313db-d382-4cc6-9f1e-34157bcee6de admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:25.246439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55866cc7-fc03-4dea-9f03-6a65cae39192 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:25.247912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55866cc7-fc03-4dea-9f03-6a65cae39192 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:25.276505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1192/2375] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:25 2022] POST /identity/v3/users => generated 286 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:25.282742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1184/2376] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:25.299668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039313db-d382-4cc6-9f1e-34157bcee6de admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:25.332476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039313db-d382-4cc6-9f1e-34157bcee6de admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:25.347229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8eba36-6d21-4e55-b0dd-25be6796bef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:25.347958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8eba36-6d21-4e55-b0dd-25be6796bef9 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:25.348239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8eba36-6d21-4e55-b0dd-25be6796bef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:25.350350 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8eba36-6d21-4e55-b0dd-25be6796bef9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:25.351517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8eba36-6d21-4e55-b0dd-25be6796bef9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:25.356641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8eba36-6d21-4e55-b0dd-25be6796bef9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:25.361125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:25.386500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1193/2377] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:56:25.392159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55866cc7-fc03-4dea-9f03-6a65cae39192 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:25.403268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55866cc7-fc03-4dea-9f03-6a65cae39192 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:25.409770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21de2d48-9671-4aa8-b001-21ef19651031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:25.410497 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21de2d48-9671-4aa8-b001-21ef19651031 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:25.410868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21de2d48-9671-4aa8-b001-21ef19651031 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:25.411268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21de2d48-9671-4aa8-b001-21ef19651031 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:25.411982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21de2d48-9671-4aa8-b001-21ef19651031 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:25.413628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21de2d48-9671-4aa8-b001-21ef19651031 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:25.430267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1185/2378] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:25.433875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8eba36-6d21-4e55-b0dd-25be6796bef9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:25.449291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8eba36-6d21-4e55-b0dd-25be6796bef9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:25.455558 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0424c15b-3f39-4e61-91d7-8b6794da1e5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:25.456425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0424c15b-3f39-4e61-91d7-8b6794da1e5c admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:25.457324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0424c15b-3f39-4e61-91d7-8b6794da1e5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:25.457774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0424c15b-3f39-4e61-91d7-8b6794da1e5c admin admin] PATH_INFO: `/v3/projects/c55edae0711d491e9db20efce20175f5/users/dac71653e3f0429fbe0224b4d8953ec0/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:25.477841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0424c15b-3f39-4e61-91d7-8b6794da1e5c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c55edae0711d491e9db20efce20175f5, user_id=dac71653e3f0429fbe0224b4d8953ec0, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:25.485680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0424c15b-3f39-4e61-91d7-8b6794da1e5c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:25.512559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1194/2379] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:25 2022] PUT /identity/v3/projects/c55edae0711d491e9db20efce20175f5/users/dac71653e3f0429fbe0224b4d8953ec0/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:25.513666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6a47ec-5052-4545-b1b9-16faba5c42d2 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:25.513961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6a47ec-5052-4545-b1b9-16faba5c42d2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:25.514623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6a47ec-5052-4545-b1b9-16faba5c42d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:25.554109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-7a6a47ec-5052-4545-b1b9-16faba5c42d2 None None] MFA Rules not processed for user `dac71653e3f0429fbe0224b4d8953ec0`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:25.555718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7a6a47ec-5052-4545-b1b9-16faba5c42d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:25.666578 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1186/2380] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:25 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:25.680586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0424c15b-3f39-4e61-91d7-8b6794da1e5c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:25.682881 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-0424c15b-3f39-4e61-91d7-8b6794da1e5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:25.776067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0424c15b-3f39-4e61-91d7-8b6794da1e5c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:25.780867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e75127-b365-4f72-bf7d-f7090f79802b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc59f80f544cef8b3869e8d37bb341', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:25.782185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e75127-b365-4f72-bf7d-f7090f79802b service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:25.782185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e75127-b365-4f72-bf7d-f7090f79802b service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:25.782546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e75127-b365-4f72-bf7d-f7090f79802b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:25.793934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e75127-b365-4f72-bf7d-f7090f79802b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:25.795601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e75127-b365-4f72-bf7d-f7090f79802b service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:25.812423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1195/2381] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:25 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:26.861246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07bd1f-53a8-4d95-8b3b-4c4b0dabde0a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:26.861831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07bd1f-53a8-4d95-8b3b-4c4b0dabde0a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:26.861831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07bd1f-53a8-4d95-8b3b-4c4b0dabde0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:26.919561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-6d07bd1f-53a8-4d95-8b3b-4c4b0dabde0a None None] MFA Rules not processed for user `6b0d44b356f84a75a1e49de5ecfef8fe`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:26.921092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6d07bd1f-53a8-4d95-8b3b-4c4b0dabde0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:26.949805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:27.010446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1187/2382] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:27.015694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e75127-b365-4f72-bf7d-f7090f79802b service glance] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:27.018670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-80e75127-b365-4f72-bf7d-f7090f79802b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:27.181739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e75127-b365-4f72-bf7d-f7090f79802b service glance] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:27.187631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b4696-0f26-4fd6-a6a6-18c668c97444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:27.188783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b4696-0f26-4fd6-a6a6-18c668c97444 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:27.189318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b4696-0f26-4fd6-a6a6-18c668c97444 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:27.189612 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2b4696-0f26-4fd6-a6a6-18c668c97444 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:27.190302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b4696-0f26-4fd6-a6a6-18c668c97444 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:27.196005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2b4696-0f26-4fd6-a6a6-18c668c97444 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:27.258148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1196/2383] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:27 2022] POST /identity/v3/projects => generated 373 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:27.262422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d07bd1f-53a8-4d95-8b3b-4c4b0dabde0a None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:27.278142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d07bd1f-53a8-4d95-8b3b-4c4b0dabde0a None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:27.401708 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4426553e-1402-4626-af22-92cc0d391362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:27.402687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4426553e-1402-4626-af22-92cc0d391362 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:27.402977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4426553e-1402-4626-af22-92cc0d391362 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:27.403254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4426553e-1402-4626-af22-92cc0d391362 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:27.404062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4426553e-1402-4626-af22-92cc0d391362 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:27.408616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4426553e-1402-4626-af22-92cc0d391362 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:27.437315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1188/2384] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:27 2022] POST /identity/v3/users => generated 293 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:27.444505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b4696-0f26-4fd6-a6a6-18c668c97444 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:27.458452 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2b4696-0f26-4fd6-a6a6-18c668c97444 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:27.466339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e55db0b-ef05-4ae3-b2ba-bb0e115230fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:27.466963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e55db0b-ef05-4ae3-b2ba-bb0e115230fc admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:27.467236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e55db0b-ef05-4ae3-b2ba-bb0e115230fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:27.467505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e55db0b-ef05-4ae3-b2ba-bb0e115230fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:27.468156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e55db0b-ef05-4ae3-b2ba-bb0e115230fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:27.469848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e55db0b-ef05-4ae3-b2ba-bb0e115230fc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:27.482691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1197/2385] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:27 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:27.487600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4426553e-1402-4626-af22-92cc0d391362 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:27.498515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4426553e-1402-4626-af22-92cc0d391362 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:27.507521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc235f0-6f83-4f2e-a819-0424e5ec2b11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:27.508206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc235f0-6f83-4f2e-a819-0424e5ec2b11 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:27.508483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc235f0-6f83-4f2e-a819-0424e5ec2b11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:27.508756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc235f0-6f83-4f2e-a819-0424e5ec2b11 admin admin] PATH_INFO: `/v3/projects/f2a3f6b63e2e4d688a076c19244314e7/users/80c49dd9d620444eab898a693bf1b3d9/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:27.538117 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc235f0-6f83-4f2e-a819-0424e5ec2b11 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2a3f6b63e2e4d688a076c19244314e7, user_id=80c49dd9d620444eab898a693bf1b3d9, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:27.540844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc235f0-6f83-4f2e-a819-0424e5ec2b11 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:27.566228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1189/2386] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:27 2022] PUT /identity/v3/projects/f2a3f6b63e2e4d688a076c19244314e7/users/80c49dd9d620444eab898a693bf1b3d9/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:56:27.570339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e55db0b-ef05-4ae3-b2ba-bb0e115230fc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:27.582283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e55db0b-ef05-4ae3-b2ba-bb0e115230fc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:27.654907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:27.672056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e62cf89-7689-44d6-822b-9f6d48235967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:27.673197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62cf89-7689-44d6-822b-9f6d48235967 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:27.673476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62cf89-7689-44d6-822b-9f6d48235967 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:27.673750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62cf89-7689-44d6-822b-9f6d48235967 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:27.674677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e62cf89-7689-44d6-822b-9f6d48235967 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:27.676031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e62cf89-7689-44d6-822b-9f6d48235967 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:27.695207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1198/2387] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:27 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:27.696532 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc235f0-6f83-4f2e-a819-0424e5ec2b11 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:27.713878 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc235f0-6f83-4f2e-a819-0424e5ec2b11 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:27.725459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f5d542-9940-4a66-a263-b4f85edd041f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:27.726211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f5d542-9940-4a66-a263-b4f85edd041f admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:27.727579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f5d542-9940-4a66-a263-b4f85edd041f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:27.727699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f5d542-9940-4a66-a263-b4f85edd041f admin admin] PATH_INFO: `/v3/projects/f2a3f6b63e2e4d688a076c19244314e7/users/80c49dd9d620444eab898a693bf1b3d9/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:27.730093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f5d542-9940-4a66-a263-b4f85edd041f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2a3f6b63e2e4d688a076c19244314e7, user_id=80c49dd9d620444eab898a693bf1b3d9, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:27.734792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f5d542-9940-4a66-a263-b4f85edd041f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:27.753640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1190/2388] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:27 2022] PUT /identity/v3/projects/f2a3f6b63e2e4d688a076c19244314e7/users/80c49dd9d620444eab898a693bf1b3d9/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:28.774651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aad2d6-28d8-478d-bf65-da1111110dad None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:28.775209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aad2d6-28d8-478d-bf65-da1111110dad None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:28.775437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aad2d6-28d8-478d-bf65-da1111110dad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:28.842801 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-82aad2d6-28d8-478d-bf65-da1111110dad None None] MFA Rules not processed for user `80c49dd9d620444eab898a693bf1b3d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:28.847644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-82aad2d6-28d8-478d-bf65-da1111110dad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:28.969626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1199/2389] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:28 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:28.981972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f5d542-9940-4a66-a263-b4f85edd041f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:28.983771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-52f5d542-9940-4a66-a263-b4f85edd041f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:29.085959 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f5d542-9940-4a66-a263-b4f85edd041f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:29.090623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1228a95-f40c-47fd-a3d2-b9010df8a883 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:29.091466 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1228a95-f40c-47fd-a3d2-b9010df8a883 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:29.091925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1228a95-f40c-47fd-a3d2-b9010df8a883 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:29.092501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1228a95-f40c-47fd-a3d2-b9010df8a883 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:29.152195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:29.230563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1228a95-f40c-47fd-a3d2-b9010df8a883 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:29.232041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1228a95-f40c-47fd-a3d2-b9010df8a883 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:29.252360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1191/2390] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:28 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:29.557982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aad2d6-28d8-478d-bf65-da1111110dad None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:29.559534 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-82aad2d6-28d8-478d-bf65-da1111110dad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:29.672682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aad2d6-28d8-478d-bf65-da1111110dad None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:29.681712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:29.682842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:29.682842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:29.683203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] PATH_INFO: `/v3/users/aaccd3976d52438e9dc639ce7d0a26f0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:29.684979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aaccd3976d52438e9dc639ce7d0a26f0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:29.686904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:29.799583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aaccd3976d52438e9dc639ce7d0a26f0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:29.806900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aaccd3976d52438e9dc639ce7d0a26f0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:29.807724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aaccd3976d52438e9dc639ce7d0a26f0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:29.808421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aaccd3976d52438e9dc639ce7d0a26f0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:29.817341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aaccd3976d52438e9dc639ce7d0a26f0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:29.825835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:29.838449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1200/2391] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:29 2022] DELETE /identity/v3/users/aaccd3976d52438e9dc639ce7d0a26f0 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:29.843141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2e1916-8eef-4cb0-856d-d39dc465ac0f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:29.843471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2e1916-8eef-4cb0-856d-d39dc465ac0f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:29.843767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2e1916-8eef-4cb0-856d-d39dc465ac0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:29.863859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-fc2e1916-8eef-4cb0-856d-d39dc465ac0f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:29.865041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fc2e1916-8eef-4cb0-856d-d39dc465ac0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:29.922965 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1192/2392] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:29.932525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:29.933899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:29.990173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb64288-ecee-49d5-9f45-1459bc789fa0 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:29.993701 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbabf0e-b9f6-4f5b-bd5b-aabc9178b365 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:29.994335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbabf0e-b9f6-4f5b-bd5b-aabc9178b365 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:29.994606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbabf0e-b9f6-4f5b-bd5b-aabc9178b365 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:29.994875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbabf0e-b9f6-4f5b-bd5b-aabc9178b365 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:30.004957 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbabf0e-b9f6-4f5b-bd5b-aabc9178b365 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:30.006438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbabf0e-b9f6-4f5b-bd5b-aabc9178b365 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:30.015651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1201/2393] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:30.479421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2e1916-8eef-4cb0-856d-d39dc465ac0f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:30.481853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fc2e1916-8eef-4cb0-856d-d39dc465ac0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:30.508374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2e1916-8eef-4cb0-856d-d39dc465ac0f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:30.526086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:30.526086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:30.526086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:30.526086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] PATH_INFO: `/v3/projects/08f05a4e059c41f2b889de30b8b78a93` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:30.526766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=08f05a4e059c41f2b889de30b8b78a93)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:30.526766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:30.590370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08f05a4e059c41f2b889de30b8b78a93', 'request_id': 'req-c7c59df6-d048-468e-9740-88d4ba2f09dd'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:30.601652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08f05a4e059c41f2b889de30b8b78a93', 'request_id': 'req-c7c59df6-d048-468e-9740-88d4ba2f09dd'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:30.608188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08f05a4e059c41f2b889de30b8b78a93', 'request_id': 'req-c7c59df6-d048-468e-9740-88d4ba2f09dd'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:30.623816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] The token cache is being invalidate because project 08f05a4e059c41f2b889de30b8b78a93 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:30.624111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:30.624824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:30.626413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1193/2394] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:30 2022] DELETE /identity/v3/projects/08f05a4e059c41f2b889de30b8b78a93 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:30.631407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83240e4e-ecfc-450e-a66a-946185844163 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:30.631727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83240e4e-ecfc-450e-a66a-946185844163 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:30.632014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83240e4e-ecfc-450e-a66a-946185844163 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:30.654712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-83240e4e-ecfc-450e-a66a-946185844163 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:30.656087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-83240e4e-ecfc-450e-a66a-946185844163 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:30.732453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:30.733928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:30.738874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1202/2395] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:30.743817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f01d39-8883-4547-8efd-41f289a4abee None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:30.744072 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f01d39-8883-4547-8efd-41f289a4abee None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:30.744344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f01d39-8883-4547-8efd-41f289a4abee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:30.771652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b6f01d39-8883-4547-8efd-41f289a4abee None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:30.773836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b6f01d39-8883-4547-8efd-41f289a4abee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:30.784020 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1203/2396] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:30.788449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f01d39-8883-4547-8efd-41f289a4abee None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:30.817554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f01d39-8883-4547-8efd-41f289a4abee None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:30.828618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c59df6-d048-468e-9740-88d4ba2f09dd admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:30.832287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa68f2b-4c0d-45e2-a97a-15adfc99f219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:30.832908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa68f2b-4c0d-45e2-a97a-15adfc99f219 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:30.833312 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa68f2b-4c0d-45e2-a97a-15adfc99f219 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:30.833583 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa68f2b-4c0d-45e2-a97a-15adfc99f219 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:30.834328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa68f2b-4c0d-45e2-a97a-15adfc99f219 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:30.834626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa68f2b-4c0d-45e2-a97a-15adfc99f219 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:30.835324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef7a5ec-db80-429a-b076-7345bc882802 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:30.836269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef7a5ec-db80-429a-b076-7345bc882802 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:30.836727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef7a5ec-db80-429a-b076-7345bc882802 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:30.837225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef7a5ec-db80-429a-b076-7345bc882802 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:30.838192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa68f2b-4c0d-45e2-a97a-15adfc99f219 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:30.840175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-bef7a5ec-db80-429a-b076-7345bc882802 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:30.869578 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1204/2397] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:56:30.873631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa68f2b-4c0d-45e2-a97a-15adfc99f219 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:30.878433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:30.887298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa68f2b-4c0d-45e2-a97a-15adfc99f219 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:30.895174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b6a5cc-8057-4455-9325-5498cc520701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:30.895806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6a5cc-8057-4455-9325-5498cc520701 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:30.896085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6a5cc-8057-4455-9325-5498cc520701 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:30.896459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6a5cc-8057-4455-9325-5498cc520701 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:30.897155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b6a5cc-8057-4455-9325-5498cc520701 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:30.902980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b6a5cc-8057-4455-9325-5498cc520701 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:30.957182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1205/2398] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:30 2022] POST /identity/v3/projects => generated 385 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:30.961180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b6a5cc-8057-4455-9325-5498cc520701 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:30.987914 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef7a5ec-db80-429a-b076-7345bc882802 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:30.989678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef7a5ec-db80-429a-b076-7345bc882802 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:31.010718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1194/2399] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:30 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:31.050893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:31.071091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b6a5cc-8057-4455-9325-5498cc520701 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:31.076976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f6058c-e253-4873-87d0-ecf68a8ab057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:31.077585 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f6058c-e253-4873-87d0-ecf68a8ab057 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:31.077859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f6058c-e253-4873-87d0-ecf68a8ab057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:31.078160 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f6058c-e253-4873-87d0-ecf68a8ab057 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:31.078894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f6058c-e253-4873-87d0-ecf68a8ab057 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:31.080791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f6058c-e253-4873-87d0-ecf68a8ab057 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:31.105920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1206/2400] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:30 2022] POST /identity/v3/users => generated 293 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:56:31.110306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef7a5ec-db80-429a-b076-7345bc882802 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:31.123191 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef7a5ec-db80-429a-b076-7345bc882802 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:31.128383 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dc6706-540d-4fc5-b2da-f69f15f8f54d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:31.129362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dc6706-540d-4fc5-b2da-f69f15f8f54d admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:31.129362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dc6706-540d-4fc5-b2da-f69f15f8f54d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:31.129547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dc6706-540d-4fc5-b2da-f69f15f8f54d admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:31.130202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dc6706-540d-4fc5-b2da-f69f15f8f54d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:31.131858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dc6706-540d-4fc5-b2da-f69f15f8f54d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:31.147147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1195/2401] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:31.150888 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f6058c-e253-4873-87d0-ecf68a8ab057 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:31.161665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f6058c-e253-4873-87d0-ecf68a8ab057 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:31.166279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50ec533-2bb6-4439-b01e-c636cef77baa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:31.166995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50ec533-2bb6-4439-b01e-c636cef77baa admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:31.167307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50ec533-2bb6-4439-b01e-c636cef77baa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:31.167575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50ec533-2bb6-4439-b01e-c636cef77baa admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:31.168422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50ec533-2bb6-4439-b01e-c636cef77baa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:31.169946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50ec533-2bb6-4439-b01e-c636cef77baa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:31.188088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1207/2402] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:31 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:31.189334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dc6706-540d-4fc5-b2da-f69f15f8f54d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:31.202149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dc6706-540d-4fc5-b2da-f69f15f8f54d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:31.212027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bef2b1-333f-47cd-84bf-f2156cff42ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:31.212931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bef2b1-333f-47cd-84bf-f2156cff42ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:31.213241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bef2b1-333f-47cd-84bf-f2156cff42ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:31.213463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bef2b1-333f-47cd-84bf-f2156cff42ab admin admin] PATH_INFO: `/v3/projects/9ccd0f3783f64e2d8732c2daab57c8b8/users/f7efb01704254d2196358b18f00d0135/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:31.232310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bef2b1-333f-47cd-84bf-f2156cff42ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ccd0f3783f64e2d8732c2daab57c8b8, user_id=f7efb01704254d2196358b18f00d0135, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:31.235023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bef2b1-333f-47cd-84bf-f2156cff42ab admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:31.254738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1196/2403] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:31 2022] PUT /identity/v3/projects/9ccd0f3783f64e2d8732c2daab57c8b8/users/f7efb01704254d2196358b18f00d0135/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:31.272521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1026e81-525f-4cfa-9a3b-1763078e52d2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:31.272771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1026e81-525f-4cfa-9a3b-1763078e52d2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:31.273047 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1026e81-525f-4cfa-9a3b-1763078e52d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:31.302943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e1026e81-525f-4cfa-9a3b-1763078e52d2 None None] MFA Rules not processed for user `f7efb01704254d2196358b18f00d0135`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:31.304164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e1026e81-525f-4cfa-9a3b-1763078e52d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:31.369506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1208/2404] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:31 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:31.381218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bef2b1-333f-47cd-84bf-f2156cff42ab admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:31.442983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bef2b1-333f-47cd-84bf-f2156cff42ab admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:31.446939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdd2e25-8c48-4798-b99e-94032528e5a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:31.447549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd2e25-8c48-4798-b99e-94032528e5a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:31.447811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd2e25-8c48-4798-b99e-94032528e5a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:31.448124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdd2e25-8c48-4798-b99e-94032528e5a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:31.458016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdd2e25-8c48-4798-b99e-94032528e5a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:31.459466 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdd2e25-8c48-4798-b99e-94032528e5a0 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:31.469034 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1197/2405] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:31 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:34.117388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1026e81-525f-4cfa-9a3b-1763078e52d2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:34.209089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1026e81-525f-4cfa-9a3b-1763078e52d2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:34.213805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:34.214560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:34.215059 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:34.215520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] PATH_INFO: `/v3/users/f7efb01704254d2196358b18f00d0135` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:34.216967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=f7efb01704254d2196358b18f00d0135)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:34.219031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:34.272115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f7efb01704254d2196358b18f00d0135'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:34.279482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f7efb01704254d2196358b18f00d0135'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:34.280661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f7efb01704254d2196358b18f00d0135'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:34.281271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f7efb01704254d2196358b18f00d0135'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:34.295472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f7efb01704254d2196358b18f00d0135'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:34.299369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:34.323028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1209/2406] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:34 2022] DELETE /identity/v3/users/f7efb01704254d2196358b18f00d0135 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:34.327930 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca64dee4-d1b7-4e69-ad01-cad4419458ce None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:34.329273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca64dee4-d1b7-4e69-ad01-cad4419458ce None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:34.329690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca64dee4-d1b7-4e69-ad01-cad4419458ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:34.350951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ca64dee4-d1b7-4e69-ad01-cad4419458ce None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:34.352388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ca64dee4-d1b7-4e69-ad01-cad4419458ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:34.421495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1198/2407] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:34.431381 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:34.432988 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:34.495247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe6e78e-4892-4c89-957c-670e092d9d5e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:34.498866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8aa7cce-4890-438e-a086-d849cedb9d3e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:34.499477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8aa7cce-4890-438e-a086-d849cedb9d3e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:34.499750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8aa7cce-4890-438e-a086-d849cedb9d3e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:34.500015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8aa7cce-4890-438e-a086-d849cedb9d3e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:34.509382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8aa7cce-4890-438e-a086-d849cedb9d3e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:34.510835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8aa7cce-4890-438e-a086-d849cedb9d3e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:34.520655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1210/2408] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:34.801926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca64dee4-d1b7-4e69-ad01-cad4419458ce None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:34.803546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ca64dee4-d1b7-4e69-ad01-cad4419458ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:34.816074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca64dee4-d1b7-4e69-ad01-cad4419458ce None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:34.820749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:34.821451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:34.821798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:34.822203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] PATH_INFO: `/v3/projects/9ccd0f3783f64e2d8732c2daab57c8b8` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:34.823396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ccd0f3783f64e2d8732c2daab57c8b8)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:34.825362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:34.840392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:34.885512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ccd0f3783f64e2d8732c2daab57c8b8', 'request_id': 'req-f174e92c-6079-420f-9965-f178d109fb92'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:34.891992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ccd0f3783f64e2d8732c2daab57c8b8', 'request_id': 'req-f174e92c-6079-420f-9965-f178d109fb92'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:34.898729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ccd0f3783f64e2d8732c2daab57c8b8', 'request_id': 'req-f174e92c-6079-420f-9965-f178d109fb92'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:34.909904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] The token cache is being invalidate because project 9ccd0f3783f64e2d8732c2daab57c8b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:34.910244 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:34.910905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f174e92c-6079-420f-9965-f178d109fb92 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:34.912386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1199/2409] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:34 2022] DELETE /identity/v3/projects/9ccd0f3783f64e2d8732c2daab57c8b8 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:43.932680 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2728686-b62a-4c54-a722-9297fcb504fd None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:43.933214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2728686-b62a-4c54-a722-9297fcb504fd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:43.933214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2728686-b62a-4c54-a722-9297fcb504fd None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:43.934574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1211/2410] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:56:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:56:43.963542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1502e1-3f23-4464-9c21-57b2cd36c51c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:43.964123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1502e1-3f23-4464-9c21-57b2cd36c51c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:43.964123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1502e1-3f23-4464-9c21-57b2cd36c51c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:43.984203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-4e1502e1-3f23-4464-9c21-57b2cd36c51c None None] MFA Rules not processed for user `5eeff4f989ae4338a182bfdadd3a8b63`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:43.986605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4e1502e1-3f23-4464-9c21-57b2cd36c51c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:44.097043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1200/2411] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:56:43 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:44.109994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2728686-b62a-4c54-a722-9297fcb504fd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:44.111758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d2728686-b62a-4c54-a722-9297fcb504fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:44.144121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2728686-b62a-4c54-a722-9297fcb504fd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:44.157026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e144ff-cf7e-48af-853d-9255a5fa9f8f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:44.157498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e144ff-cf7e-48af-853d-9255a5fa9f8f service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:44.157775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e144ff-cf7e-48af-853d-9255a5fa9f8f service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:44.158034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e144ff-cf7e-48af-853d-9255a5fa9f8f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:44.173357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e144ff-cf7e-48af-853d-9255a5fa9f8f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:44.173357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e144ff-cf7e-48af-853d-9255a5fa9f8f service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:44.179746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1212/2412] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:44.579533 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1502e1-3f23-4464-9c21-57b2cd36c51c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:44.581000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4e1502e1-3f23-4464-9c21-57b2cd36c51c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:44.655492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1502e1-3f23-4464-9c21-57b2cd36c51c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:44.660555 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:44.661185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:44.661464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:44.661735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] PATH_INFO: `/v3/users/6b0d44b356f84a75a1e49de5ecfef8fe` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:44.662941 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b0d44b356f84a75a1e49de5ecfef8fe)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:44.664854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:44.703558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:44.713600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b0d44b356f84a75a1e49de5ecfef8fe'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:44.714619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b0d44b356f84a75a1e49de5ecfef8fe'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:44.723419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b0d44b356f84a75a1e49de5ecfef8fe'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:44.724125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b0d44b356f84a75a1e49de5ecfef8fe'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:44.730766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b0d44b356f84a75a1e49de5ecfef8fe'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:44.750541 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1201/2413] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:44 2022] DELETE /identity/v3/users/6b0d44b356f84a75a1e49de5ecfef8fe => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:44.927107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e144ff-cf7e-48af-853d-9255a5fa9f8f service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:44.937745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e144ff-cf7e-48af-853d-9255a5fa9f8f service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:45.002671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:45.003298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:45.003569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:45.003835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] PATH_INFO: `/v3/projects/e8b83d2c37b747b980257937fba802bd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:45.004917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8b83d2c37b747b980257937fba802bd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:45.006825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:45.052932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8b83d2c37b747b980257937fba802bd', 'request_id': 'req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.059507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8b83d2c37b747b980257937fba802bd', 'request_id': 'req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.066422 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8b83d2c37b747b980257937fba802bd', 'request_id': 'req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.073532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:45.078092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] The token cache is being invalidate because project e8b83d2c37b747b980257937fba802bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:45.078632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.079747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.081356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1213/2414] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:44 2022] DELETE /identity/v3/projects/e8b83d2c37b747b980257937fba802bd => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:45.085448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:45.086925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:45.100139 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cbdf4d-468c-4d30-aefe-e00153ffa05c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:45.156855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:45.157403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:45.157807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:45.158170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] PATH_INFO: `/v3/users/80c49dd9d620444eab898a693bf1b3d9` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:45.159355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80c49dd9d620444eab898a693bf1b3d9)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:45.161296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:45.209443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80c49dd9d620444eab898a693bf1b3d9'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.210103 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80c49dd9d620444eab898a693bf1b3d9'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.217045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80c49dd9d620444eab898a693bf1b3d9'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.217671 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80c49dd9d620444eab898a693bf1b3d9'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.227282 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-416a1ec8-b3f8-411c-aa8c-9fe60c4f17c6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80c49dd9d620444eab898a693bf1b3d9'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.245550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1202/2415] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:45 2022] DELETE /identity/v3/users/80c49dd9d620444eab898a693bf1b3d9 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:56:45.417305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:45.418800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:45.435404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8dc7a0-7afc-4eed-b8d5-5778e437d8b2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:45.495769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:45.497298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:45.497298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:45.497298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] PATH_INFO: `/v3/projects/f2a3f6b63e2e4d688a076c19244314e7` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:45.498246 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2a3f6b63e2e4d688a076c19244314e7)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:45.500328 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:45.557888 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2a3f6b63e2e4d688a076c19244314e7', 'request_id': 'req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.566615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2a3f6b63e2e4d688a076c19244314e7', 'request_id': 'req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.572931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2a3f6b63e2e4d688a076c19244314e7', 'request_id': 'req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.583663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] The token cache is being invalidate because project f2a3f6b63e2e4d688a076c19244314e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:45.583948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.584590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a5a2fcd2-9fbe-4c9e-a732-bdd1b74ce550 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:45.586050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1214/2416] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:45 2022] DELETE /identity/v3/projects/f2a3f6b63e2e4d688a076c19244314e7 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:45.591315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e86e7-20dc-4223-b0d5-aec377ea5151 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:45.591611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e86e7-20dc-4223-b0d5-aec377ea5151 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:45.591894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e86e7-20dc-4223-b0d5-aec377ea5151 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:45.615448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-691e86e7-20dc-4223-b0d5-aec377ea5151 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:45.616678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-691e86e7-20dc-4223-b0d5-aec377ea5151 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:45.644408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:45.681801 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1203/2417] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:45.689940 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca9b68-8c68-4ea1-9f87-4d04fd8a7ef2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:45.690310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca9b68-8c68-4ea1-9f87-4d04fd8a7ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:45.690644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca9b68-8c68-4ea1-9f87-4d04fd8a7ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:45.714146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6bca9b68-8c68-4ea1-9f87-4d04fd8a7ef2 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:45.715441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6bca9b68-8c68-4ea1-9f87-4d04fd8a7ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:45.723292 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1215/2418] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:45.728919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691e86e7-20dc-4223-b0d5-aec377ea5151 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:45.740686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691e86e7-20dc-4223-b0d5-aec377ea5151 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:45.747204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add80bb0-6308-4f9f-baef-5bf6ce0cd85f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:45.748291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add80bb0-6308-4f9f-baef-5bf6ce0cd85f admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:45.748569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add80bb0-6308-4f9f-baef-5bf6ce0cd85f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:45.748844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add80bb0-6308-4f9f-baef-5bf6ce0cd85f admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:45.749499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add80bb0-6308-4f9f-baef-5bf6ce0cd85f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:45.749806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add80bb0-6308-4f9f-baef-5bf6ce0cd85f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:45.751499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add80bb0-6308-4f9f-baef-5bf6ce0cd85f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:45.763886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1204/2419] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:56:45.768820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca9b68-8c68-4ea1-9f87-4d04fd8a7ef2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:45.778435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca9b68-8c68-4ea1-9f87-4d04fd8a7ef2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:45.783609 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45d761b-b598-4902-9bd9-19adde9eb6b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:45.784229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d761b-b598-4902-9bd9-19adde9eb6b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:45.784506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d761b-b598-4902-9bd9-19adde9eb6b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:45.784776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45d761b-b598-4902-9bd9-19adde9eb6b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:45.785411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45d761b-b598-4902-9bd9-19adde9eb6b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:45.787341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45d761b-b598-4902-9bd9-19adde9eb6b1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:45.816977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1216/2420] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:45 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:45.821150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add80bb0-6308-4f9f-baef-5bf6ce0cd85f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:45.831599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add80bb0-6308-4f9f-baef-5bf6ce0cd85f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:45.895373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470cad25-5262-4593-a631-e7c717a763ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:45.896011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470cad25-5262-4593-a631-e7c717a763ee admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:45.896286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470cad25-5262-4593-a631-e7c717a763ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:45.896552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470cad25-5262-4593-a631-e7c717a763ee admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:45.897170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470cad25-5262-4593-a631-e7c717a763ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:45.899172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470cad25-5262-4593-a631-e7c717a763ee admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:45.920191 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1205/2421] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:45 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:45.924447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45d761b-b598-4902-9bd9-19adde9eb6b1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:45.934655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45d761b-b598-4902-9bd9-19adde9eb6b1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:45.939706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223559c7-cc8d-4b07-be6d-9d5bd5237a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:45.940279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223559c7-cc8d-4b07-be6d-9d5bd5237a01 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:45.940552 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223559c7-cc8d-4b07-be6d-9d5bd5237a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:45.940816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223559c7-cc8d-4b07-be6d-9d5bd5237a01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:45.941442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223559c7-cc8d-4b07-be6d-9d5bd5237a01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:45.943054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223559c7-cc8d-4b07-be6d-9d5bd5237a01 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:45.952667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1217/2422] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:45 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:45.957227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470cad25-5262-4593-a631-e7c717a763ee admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:45.967436 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470cad25-5262-4593-a631-e7c717a763ee admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:45.972278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854703cf-e1ed-4eab-8cec-c2d4d668586d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:45.972881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854703cf-e1ed-4eab-8cec-c2d4d668586d admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:45.973163 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854703cf-e1ed-4eab-8cec-c2d4d668586d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:45.973440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854703cf-e1ed-4eab-8cec-c2d4d668586d admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:45.974115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854703cf-e1ed-4eab-8cec-c2d4d668586d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:45.975724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854703cf-e1ed-4eab-8cec-c2d4d668586d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:45.989379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1206/2423] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:45.990711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223559c7-cc8d-4b07-be6d-9d5bd5237a01 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:46.001129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223559c7-cc8d-4b07-be6d-9d5bd5237a01 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:46.005987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab51b361-6aa0-4bee-890e-4c3dd9911a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:46.006800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab51b361-6aa0-4bee-890e-4c3dd9911a03 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.007114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab51b361-6aa0-4bee-890e-4c3dd9911a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.007478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab51b361-6aa0-4bee-890e-4c3dd9911a03 admin admin] PATH_INFO: `/v3/projects/48afba6d3001437f88089425d7924297/users/400035b0ac414fed8f3de6e250377ad0/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.023925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab51b361-6aa0-4bee-890e-4c3dd9911a03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48afba6d3001437f88089425d7924297, user_id=400035b0ac414fed8f3de6e250377ad0, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:46.026942 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab51b361-6aa0-4bee-890e-4c3dd9911a03 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:46.041188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1218/2424] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:45 2022] PUT /identity/v3/projects/48afba6d3001437f88089425d7924297/users/400035b0ac414fed8f3de6e250377ad0/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:46.108451 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a4aae0-0af9-430a-8ead-f2aa5cb01da3 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.109022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a4aae0-0af9-430a-8ead-f2aa5cb01da3 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.109022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a4aae0-0af9-430a-8ead-f2aa5cb01da3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.138824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-89a4aae0-0af9-430a-8ead-f2aa5cb01da3 None None] MFA Rules not processed for user `400035b0ac414fed8f3de6e250377ad0`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:46.140016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-89a4aae0-0af9-430a-8ead-f2aa5cb01da3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:46.205168 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1207/2425] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:46.209643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab51b361-6aa0-4bee-890e-4c3dd9911a03 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:46.239869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:46.279021 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab51b361-6aa0-4bee-890e-4c3dd9911a03 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:46.283573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956fde2b-e159-447a-b0c0-786d58a6a2bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:46.284162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956fde2b-e159-447a-b0c0-786d58a6a2bb admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.284494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956fde2b-e159-447a-b0c0-786d58a6a2bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.284766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956fde2b-e159-447a-b0c0-786d58a6a2bb admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.285391 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956fde2b-e159-447a-b0c0-786d58a6a2bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:46.287306 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956fde2b-e159-447a-b0c0-786d58a6a2bb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:46.313440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1219/2426] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:46 2022] POST /identity/v3/projects => generated 383 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:46.317674 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a4aae0-0af9-430a-8ead-f2aa5cb01da3 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:46.328665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a4aae0-0af9-430a-8ead-f2aa5cb01da3 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:46.351811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:46.384333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467f75c5-6a98-4ce5-8fb1-bd79a2403cf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:46.384939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467f75c5-6a98-4ce5-8fb1-bd79a2403cf0 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.385218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467f75c5-6a98-4ce5-8fb1-bd79a2403cf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.385487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467f75c5-6a98-4ce5-8fb1-bd79a2403cf0 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.386147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467f75c5-6a98-4ce5-8fb1-bd79a2403cf0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:46.388030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467f75c5-6a98-4ce5-8fb1-bd79a2403cf0 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:46.416586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1208/2427] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:46 2022] POST /identity/v3/users => generated 292 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:46.420595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956fde2b-e159-447a-b0c0-786d58a6a2bb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:46.431131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956fde2b-e159-447a-b0c0-786d58a6a2bb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:46.436055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ff2bf0-b870-4151-857a-c18155012c84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:46.436570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ff2bf0-b870-4151-857a-c18155012c84 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.436843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ff2bf0-b870-4151-857a-c18155012c84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.437107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ff2bf0-b870-4151-857a-c18155012c84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.437769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ff2bf0-b870-4151-857a-c18155012c84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:46.439397 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ff2bf0-b870-4151-857a-c18155012c84 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:46.450296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1220/2428] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:46.454829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467f75c5-6a98-4ce5-8fb1-bd79a2403cf0 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:46.464834 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467f75c5-6a98-4ce5-8fb1-bd79a2403cf0 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:46.470051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7995f6-e2d0-41fc-aab5-9c0a2b91e74c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:46.470675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7995f6-e2d0-41fc-aab5-9c0a2b91e74c admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.470947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7995f6-e2d0-41fc-aab5-9c0a2b91e74c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.471235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7995f6-e2d0-41fc-aab5-9c0a2b91e74c admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.471868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7995f6-e2d0-41fc-aab5-9c0a2b91e74c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:46.473446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7995f6-e2d0-41fc-aab5-9c0a2b91e74c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:46.483518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1209/2429] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:46.487481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ff2bf0-b870-4151-857a-c18155012c84 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:46.497593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ff2bf0-b870-4151-857a-c18155012c84 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:46.502638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03429cd-baac-4dcf-a3ff-5c09e4471fe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:46.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03429cd-baac-4dcf-a3ff-5c09e4471fe1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.503261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03429cd-baac-4dcf-a3ff-5c09e4471fe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.503565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03429cd-baac-4dcf-a3ff-5c09e4471fe1 admin admin] PATH_INFO: `/v3/projects/792aa143d0414b3184b731ac6b88b84d/users/211a9cf42ecb49e98db55831dce56e3c/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.533326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03429cd-baac-4dcf-a3ff-5c09e4471fe1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=792aa143d0414b3184b731ac6b88b84d, user_id=211a9cf42ecb49e98db55831dce56e3c, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:46.536022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03429cd-baac-4dcf-a3ff-5c09e4471fe1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:46.549842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1221/2430] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:46 2022] PUT /identity/v3/projects/792aa143d0414b3184b731ac6b88b84d/users/211a9cf42ecb49e98db55831dce56e3c/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:46.560325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98a7943-4695-493f-b14d-6fc11809ad9e None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.560737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98a7943-4695-493f-b14d-6fc11809ad9e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.561099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98a7943-4695-493f-b14d-6fc11809ad9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.596403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-b98a7943-4695-493f-b14d-6fc11809ad9e None None] MFA Rules not processed for user `211a9cf42ecb49e98db55831dce56e3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:46.597650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b98a7943-4695-493f-b14d-6fc11809ad9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:46.675929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1210/2431] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:46 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:46.677042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03429cd-baac-4dcf-a3ff-5c09e4471fe1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:46.766378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03429cd-baac-4dcf-a3ff-5c09e4471fe1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:46.773953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54eb1b84-e093-4b19-b7d8-204195e35322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:46.775383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54eb1b84-e093-4b19-b7d8-204195e35322 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.775838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54eb1b84-e093-4b19-b7d8-204195e35322 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.785476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54eb1b84-e093-4b19-b7d8-204195e35322 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.786243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54eb1b84-e093-4b19-b7d8-204195e35322 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:46.788196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54eb1b84-e093-4b19-b7d8-204195e35322 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:46.818269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1222/2432] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:46 2022] POST /identity/v3/projects => generated 379 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:46.822780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98a7943-4695-493f-b14d-6fc11809ad9e None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:46.835133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98a7943-4695-493f-b14d-6fc11809ad9e None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:46.920399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11dacb9-08a5-48cd-9eeb-710d9ee45700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:46.921025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11dacb9-08a5-48cd-9eeb-710d9ee45700 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.921323 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11dacb9-08a5-48cd-9eeb-710d9ee45700 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.921915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11dacb9-08a5-48cd-9eeb-710d9ee45700 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.922351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11dacb9-08a5-48cd-9eeb-710d9ee45700 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:46.924336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11dacb9-08a5-48cd-9eeb-710d9ee45700 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:46.960683 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1211/2433] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:46 2022] POST /identity/v3/users => generated 296 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:46.964540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54eb1b84-e093-4b19-b7d8-204195e35322 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:46.975217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54eb1b84-e093-4b19-b7d8-204195e35322 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:46.980294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75708542-f39f-4840-960f-a4f0e899727f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:46.980938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75708542-f39f-4840-960f-a4f0e899727f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:46.981213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75708542-f39f-4840-960f-a4f0e899727f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:46.981502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75708542-f39f-4840-960f-a4f0e899727f admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:46.982155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75708542-f39f-4840-960f-a4f0e899727f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:46.983804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75708542-f39f-4840-960f-a4f0e899727f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:46.995907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1223/2434] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:46 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:47.000104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11dacb9-08a5-48cd-9eeb-710d9ee45700 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:47.010325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11dacb9-08a5-48cd-9eeb-710d9ee45700 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:47.014978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a726f53-3ffd-4cd9-b03f-46b54c298804 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:47.016365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a726f53-3ffd-4cd9-b03f-46b54c298804 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:47.016640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a726f53-3ffd-4cd9-b03f-46b54c298804 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:47.016919 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a726f53-3ffd-4cd9-b03f-46b54c298804 admin admin] PATH_INFO: `/v3/projects/39749481009d4849882156805e97029b/users/a182dcde927e44b78ae8d958796723c6/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:47.033784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a726f53-3ffd-4cd9-b03f-46b54c298804 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39749481009d4849882156805e97029b, user_id=a182dcde927e44b78ae8d958796723c6, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:47.036496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a726f53-3ffd-4cd9-b03f-46b54c298804 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:47.048783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:47.056932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1212/2435] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:46 2022] PUT /identity/v3/projects/39749481009d4849882156805e97029b/users/a182dcde927e44b78ae8d958796723c6/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:47.060628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75708542-f39f-4840-960f-a4f0e899727f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:47.072958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75708542-f39f-4840-960f-a4f0e899727f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:47.122477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:47.134674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1512de9-b1db-49b2-9249-cb82ff2426fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:47.135283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1512de9-b1db-49b2-9249-cb82ff2426fe admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:47.135559 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1512de9-b1db-49b2-9249-cb82ff2426fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:47.135828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1512de9-b1db-49b2-9249-cb82ff2426fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:47.136486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1512de9-b1db-49b2-9249-cb82ff2426fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:47.138111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1512de9-b1db-49b2-9249-cb82ff2426fe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:47.148392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1224/2436] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:47 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:47.153677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a726f53-3ffd-4cd9-b03f-46b54c298804 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:47.165797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a726f53-3ffd-4cd9-b03f-46b54c298804 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:47.170586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1c138-2ae5-43c8-a315-601344e63a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:47.171345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1c138-2ae5-43c8-a315-601344e63a95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:47.172325 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1c138-2ae5-43c8-a315-601344e63a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:47.172609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1c138-2ae5-43c8-a315-601344e63a95 admin admin] PATH_INFO: `/v3/projects/39749481009d4849882156805e97029b/users/a182dcde927e44b78ae8d958796723c6/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:47.174722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d1c138-2ae5-43c8-a315-601344e63a95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39749481009d4849882156805e97029b, user_id=a182dcde927e44b78ae8d958796723c6, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:47.177640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d1c138-2ae5-43c8-a315-601344e63a95 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:47.189761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1213/2437] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:47 2022] PUT /identity/v3/projects/39749481009d4849882156805e97029b/users/a182dcde927e44b78ae8d958796723c6/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:47.202727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0da9b6-9930-42bf-b7b6-015524f43428 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:47.203285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0da9b6-9930-42bf-b7b6-015524f43428 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:47.203581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0da9b6-9930-42bf-b7b6-015524f43428 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:47.252924 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6f0da9b6-9930-42bf-b7b6-015524f43428 None None] MFA Rules not processed for user `a182dcde927e44b78ae8d958796723c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:47.255102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6f0da9b6-9930-42bf-b7b6-015524f43428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:47.332353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1225/2438] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:47 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:47.344057 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1c138-2ae5-43c8-a315-601344e63a95 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:47.346229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-26d1c138-2ae5-43c8-a315-601344e63a95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:47.410292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1c138-2ae5-43c8-a315-601344e63a95 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:47.413977 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922059c3-3c51-4407-80d0-90199c5d5c31 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:47.414620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922059c3-3c51-4407-80d0-90199c5d5c31 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:47.414895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922059c3-3c51-4407-80d0-90199c5d5c31 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:47.415164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922059c3-3c51-4407-80d0-90199c5d5c31 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:47.473506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922059c3-3c51-4407-80d0-90199c5d5c31 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:47.474869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922059c3-3c51-4407-80d0-90199c5d5c31 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:47.483699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1214/2439] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:47 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:56:49.009791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0da9b6-9930-42bf-b7b6-015524f43428 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:49.011846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6f0da9b6-9930-42bf-b7b6-015524f43428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:49.025976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0da9b6-9930-42bf-b7b6-015524f43428 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:49.029859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff08b757-4f29-4dac-904a-933a63ae8ea8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:49.030620 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff08b757-4f29-4dac-904a-933a63ae8ea8 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:49.031508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff08b757-4f29-4dac-904a-933a63ae8ea8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:49.031960 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff08b757-4f29-4dac-904a-933a63ae8ea8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:49.109374 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff08b757-4f29-4dac-904a-933a63ae8ea8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:49.110849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff08b757-4f29-4dac-904a-933a63ae8ea8 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:49.119657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1226/2440] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:49.691637 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922059c3-3c51-4407-80d0-90199c5d5c31 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:49.700526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922059c3-3c51-4407-80d0-90199c5d5c31 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:49.705065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a1bb49-ac0e-4e00-9c20-280be4b76954 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:49.705665 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a1bb49-ac0e-4e00-9c20-280be4b76954 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:49.706898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a1bb49-ac0e-4e00-9c20-280be4b76954 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:49.707322 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a1bb49-ac0e-4e00-9c20-280be4b76954 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:49.717789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a1bb49-ac0e-4e00-9c20-280be4b76954 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:49.719721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a1bb49-ac0e-4e00-9c20-280be4b76954 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:49.730452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1215/2441] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:49 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:54.018016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff08b757-4f29-4dac-904a-933a63ae8ea8 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:54.098371 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff08b757-4f29-4dac-904a-933a63ae8ea8 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:54.103801 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:54.104945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:54.105226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:54.105500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] PATH_INFO: `/v3/users/400035b0ac414fed8f3de6e250377ad0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:54.108033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] RBAC: Authorizing `identity:delete_user(user_id=400035b0ac414fed8f3de6e250377ad0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:54.108742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:54.126420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:54.166382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '400035b0ac414fed8f3de6e250377ad0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:54.175785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '400035b0ac414fed8f3de6e250377ad0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:54.176640 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '400035b0ac414fed8f3de6e250377ad0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:54.177770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '400035b0ac414fed8f3de6e250377ad0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:54.189823 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '400035b0ac414fed8f3de6e250377ad0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:54.212741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1227/2442] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:54 2022] DELETE /identity/v3/users/400035b0ac414fed8f3de6e250377ad0 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:54.217187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981fab78-3df6-4a99-96a8-4c0b3475c174 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:54.218018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981fab78-3df6-4a99-96a8-4c0b3475c174 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:54.218357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981fab78-3df6-4a99-96a8-4c0b3475c174 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:54.240661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-981fab78-3df6-4a99-96a8-4c0b3475c174 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:54.242650 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-981fab78-3df6-4a99-96a8-4c0b3475c174 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:54.331906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1216/2443] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:54.338786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:54.340204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:54.371864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2294c2c-7313-437a-b28c-061886b5789a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:54.372195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2294c2c-7313-437a-b28c-061886b5789a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:54.372471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2294c2c-7313-437a-b28c-061886b5789a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:54.392336 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f2294c2c-7313-437a-b28c-061886b5789a None None] MFA Rules not processed for user `afe98894d83f4678bb134782d90706e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:54.393546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f2294c2c-7313-437a-b28c-061886b5789a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:54.414460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e27d511-5337-47c1-bfc9-867f356bec37 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:54.418698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996d66b3-1f3d-43fc-b7fe-208c94ec3679 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:54.419337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:54.419463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996d66b3-1f3d-43fc-b7fe-208c94ec3679 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:54.419678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996d66b3-1f3d-43fc-b7fe-208c94ec3679 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:54.419983 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996d66b3-1f3d-43fc-b7fe-208c94ec3679 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:54.432325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996d66b3-1f3d-43fc-b7fe-208c94ec3679 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:54.433768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996d66b3-1f3d-43fc-b7fe-208c94ec3679 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:54.445454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1228/2444] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:56:54.457775 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1217/2445] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:54 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:56:54.884526 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996d66b3-1f3d-43fc-b7fe-208c94ec3679 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:54.886359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-996d66b3-1f3d-43fc-b7fe-208c94ec3679 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:54.901954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996d66b3-1f3d-43fc-b7fe-208c94ec3679 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:54.908123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:54.908952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:54.909164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:54.909482 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] PATH_INFO: `/v3/projects/48afba6d3001437f88089425d7924297` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:54.910723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] RBAC: Authorizing `identity:delete_project(project_id=48afba6d3001437f88089425d7924297)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:54.912641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:54.977956 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48afba6d3001437f88089425d7924297', 'request_id': 'req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:54.989691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48afba6d3001437f88089425d7924297', 'request_id': 'req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:54.998644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48afba6d3001437f88089425d7924297', 'request_id': 'req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.011421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] The token cache is being invalidate because project 48afba6d3001437f88089425d7924297 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:55.011711 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.012478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.013973 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1229/2446] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:54 2022] DELETE /identity/v3/projects/48afba6d3001437f88089425d7924297 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:55.018293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2294c2c-7313-437a-b28c-061886b5789a None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:55.033972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2294c2c-7313-437a-b28c-061886b5789a None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:55.094588 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:55.095145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:55.095422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:55.095693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] PATH_INFO: `/v3/users/211a9cf42ecb49e98db55831dce56e3c` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:55.096839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=211a9cf42ecb49e98db55831dce56e3c)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:55.098809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:55.149837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '211a9cf42ecb49e98db55831dce56e3c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.150539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '211a9cf42ecb49e98db55831dce56e3c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.158172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '211a9cf42ecb49e98db55831dce56e3c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.158869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '211a9cf42ecb49e98db55831dce56e3c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.167305 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '211a9cf42ecb49e98db55831dce56e3c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.185443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1218/2447] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:55 2022] DELETE /identity/v3/users/211a9cf42ecb49e98db55831dce56e3c => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:55.473808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:55.475804 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:55.492241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc1e8e4-d306-4bee-a070-f660ddeb6e4c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:55.506551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:55.564450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:55.565114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:55.565414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:55.565709 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] PATH_INFO: `/v3/projects/792aa143d0414b3184b731ac6b88b84d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:55.566934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=792aa143d0414b3184b731ac6b88b84d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:55.568890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:55.621445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '792aa143d0414b3184b731ac6b88b84d', 'request_id': 'req-103abd57-950f-4431-8d40-26f0f18d1ef2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '792aa143d0414b3184b731ac6b88b84d', 'request_id': 'req-103abd57-950f-4431-8d40-26f0f18d1ef2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.637424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '792aa143d0414b3184b731ac6b88b84d', 'request_id': 'req-103abd57-950f-4431-8d40-26f0f18d1ef2'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.649691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] The token cache is being invalidate because project 792aa143d0414b3184b731ac6b88b84d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:55.650038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.650799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.652260 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1230/2448] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:55 2022] DELETE /identity/v3/projects/792aa143d0414b3184b731ac6b88b84d => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:55.656668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:55.658160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:55.670355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd646ed0-8507-4a5e-874b-735181147de1 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:55.720223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:55.731832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:55.732611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:55.732962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:55.733243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] PATH_INFO: `/v3/users/a182dcde927e44b78ae8d958796723c6` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:55.736817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a182dcde927e44b78ae8d958796723c6)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:55.740079 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:55.792259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a182dcde927e44b78ae8d958796723c6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.792842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a182dcde927e44b78ae8d958796723c6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.800598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a182dcde927e44b78ae8d958796723c6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.801231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a182dcde927e44b78ae8d958796723c6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.809592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-26dd9081-e49e-4ecb-bd9a-b62064963716 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a182dcde927e44b78ae8d958796723c6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:55.829594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1219/2449] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:56:55 2022] DELETE /identity/v3/users/a182dcde927e44b78ae8d958796723c6 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:56.173818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:56.175352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:56.188832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103abd57-950f-4431-8d40-26f0f18d1ef2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:56.252438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:56.253059 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.253337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.253604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] PATH_INFO: `/v3/projects/39749481009d4849882156805e97029b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.254735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39749481009d4849882156805e97029b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:56.256624 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:56.302661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39749481009d4849882156805e97029b', 'request_id': 'req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:56.306224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:56.313405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39749481009d4849882156805e97029b', 'request_id': 'req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:56.320918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39749481009d4849882156805e97029b', 'request_id': 'req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:56.332717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] The token cache is being invalidate because project 39749481009d4849882156805e97029b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:56:56.333105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:56.333948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4359e8dd-14c8-45f1-beaf-cc1a9fd58d84 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:56:56.335870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1231/2450] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:56:56 2022] DELETE /identity/v3/projects/39749481009d4849882156805e97029b => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:56:56.341030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d9d5cc-f10e-4ded-99f8-d9d7aadb5c97 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.341321 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d9d5cc-f10e-4ded-99f8-d9d7aadb5c97 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.341594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d9d5cc-f10e-4ded-99f8-d9d7aadb5c97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.361794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e4d9d5cc-f10e-4ded-99f8-d9d7aadb5c97 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:56.363017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e4d9d5cc-f10e-4ded-99f8-d9d7aadb5c97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:56.420178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1220/2451] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:56.427865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d282713-992e-4836-8ac2-93e71047fce1 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.428153 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d282713-992e-4836-8ac2-93e71047fce1 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.428430 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d282713-992e-4836-8ac2-93e71047fce1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.448781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9d282713-992e-4836-8ac2-93e71047fce1 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:56.450024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9d282713-992e-4836-8ac2-93e71047fce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:56.458608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1232/2452] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:56.462861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d9d5cc-f10e-4ded-99f8-d9d7aadb5c97 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:56.472676 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d9d5cc-f10e-4ded-99f8-d9d7aadb5c97 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:56.477205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7531ea-f285-4753-bcce-0966a45d17a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:56.477827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7531ea-f285-4753-bcce-0966a45d17a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.478155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7531ea-f285-4753-bcce-0966a45d17a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.478434 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7531ea-f285-4753-bcce-0966a45d17a1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.479076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7531ea-f285-4753-bcce-0966a45d17a1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:56:56.479381 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7531ea-f285-4753-bcce-0966a45d17a1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:56.480971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7531ea-f285-4753-bcce-0966a45d17a1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:56.492899 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1221/2453] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:56:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:56:56.498360 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d282713-992e-4836-8ac2-93e71047fce1 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:56.508416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d282713-992e-4836-8ac2-93e71047fce1 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:56.512975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33016675-d304-414f-a845-2df0c5182d7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:56.513564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33016675-d304-414f-a845-2df0c5182d7b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.513838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33016675-d304-414f-a845-2df0c5182d7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.514140 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33016675-d304-414f-a845-2df0c5182d7b admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.514784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33016675-d304-414f-a845-2df0c5182d7b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:56.517269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33016675-d304-414f-a845-2df0c5182d7b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:56.545132 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1233/2454] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:56:56 2022] POST /identity/v3/projects => generated 399 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:56.549933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7531ea-f285-4753-bcce-0966a45d17a1 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:56.559832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7531ea-f285-4753-bcce-0966a45d17a1 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:56.619517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec209051-3202-4873-9415-45cf988437ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:56.620392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec209051-3202-4873-9415-45cf988437ea admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.620677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec209051-3202-4873-9415-45cf988437ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.620949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec209051-3202-4873-9415-45cf988437ea admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.621582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec209051-3202-4873-9415-45cf988437ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:56.623543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec209051-3202-4873-9415-45cf988437ea admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:56.632110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:56:56.650220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1222/2455] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:56:56 2022] POST /identity/v3/users => generated 300 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:56:56.651410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33016675-d304-414f-a845-2df0c5182d7b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:56.662242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33016675-d304-414f-a845-2df0c5182d7b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:56.666843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f6d9dc-929a-4451-acf6-b88d51ac0d23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:56.667435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f6d9dc-929a-4451-acf6-b88d51ac0d23 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.667710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f6d9dc-929a-4451-acf6-b88d51ac0d23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.667976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f6d9dc-929a-4451-acf6-b88d51ac0d23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.668605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f6d9dc-929a-4451-acf6-b88d51ac0d23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:56.670255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f6d9dc-929a-4451-acf6-b88d51ac0d23 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:56.680492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1234/2456] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:56.684942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec209051-3202-4873-9415-45cf988437ea admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:56.697287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec209051-3202-4873-9415-45cf988437ea admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:56.702628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b153e839-0dc5-4ff6-856f-a434caf8fa92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:56.703245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b153e839-0dc5-4ff6-856f-a434caf8fa92 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.703524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b153e839-0dc5-4ff6-856f-a434caf8fa92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.703795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b153e839-0dc5-4ff6-856f-a434caf8fa92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.704439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b153e839-0dc5-4ff6-856f-a434caf8fa92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:56.706109 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b153e839-0dc5-4ff6-856f-a434caf8fa92 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:56.717060 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1223/2457] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:56:56.720975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f6d9dc-929a-4451-acf6-b88d51ac0d23 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:56.732398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f6d9dc-929a-4451-acf6-b88d51ac0d23 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:56.737508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8141298-c036-4cab-95c3-19bc6de7bfc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:56.738271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8141298-c036-4cab-95c3-19bc6de7bfc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.738545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8141298-c036-4cab-95c3-19bc6de7bfc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.738842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8141298-c036-4cab-95c3-19bc6de7bfc3 admin admin] PATH_INFO: `/v3/projects/b8122c30d932407898c08533de53ce7b/users/f2ac13f6d7d34a8884140d1f66bbf55b/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.755214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8141298-c036-4cab-95c3-19bc6de7bfc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8122c30d932407898c08533de53ce7b, user_id=f2ac13f6d7d34a8884140d1f66bbf55b, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:56.757891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8141298-c036-4cab-95c3-19bc6de7bfc3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:56.770285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1235/2458] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:56:56 2022] PUT /identity/v3/projects/b8122c30d932407898c08533de53ce7b/users/f2ac13f6d7d34a8884140d1f66bbf55b/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:56:56.780419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f368eb-5ac1-442f-9c37-baf9509869a3 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.780707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f368eb-5ac1-442f-9c37-baf9509869a3 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.780979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f368eb-5ac1-442f-9c37-baf9509869a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.808228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-51f368eb-5ac1-442f-9c37-baf9509869a3 None None] MFA Rules not processed for user `f2ac13f6d7d34a8884140d1f66bbf55b`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:56:56.809438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-51f368eb-5ac1-442f-9c37-baf9509869a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:56.872862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1224/2459] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:56:56.884732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8141298-c036-4cab-95c3-19bc6de7bfc3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:56:56.886448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b8141298-c036-4cab-95c3-19bc6de7bfc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:56:56.940561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8141298-c036-4cab-95c3-19bc6de7bfc3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:56:56.944283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d73bbd7-44e8-404c-b845-eb6e135360ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:56:56.944868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d73bbd7-44e8-404c-b845-eb6e135360ec service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:56:56.945142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d73bbd7-44e8-404c-b845-eb6e135360ec service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:56:56.945408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d73bbd7-44e8-404c-b845-eb6e135360ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:56:56.953975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d73bbd7-44e8-404c-b845-eb6e135360ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:56:56.955418 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d73bbd7-44e8-404c-b845-eb6e135360ec service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:56:56.963967 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1236/2460] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:56:56 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:57:05.093908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f368eb-5ac1-442f-9c37-baf9509869a3 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:05.151807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d73bbd7-44e8-404c-b845-eb6e135360ec service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:05.153305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1d73bbd7-44e8-404c-b845-eb6e135360ec service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:05.154133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f368eb-5ac1-442f-9c37-baf9509869a3 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:05.159189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:05.159819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:05.160099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:05.160373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] PATH_INFO: `/v3/users/f2ac13f6d7d34a8884140d1f66bbf55b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:05.161533 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2ac13f6d7d34a8884140d1f66bbf55b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:05.163648 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:05.170382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d73bbd7-44e8-404c-b845-eb6e135360ec service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:05.175637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:05.176253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:05.176531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:05.176810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] PATH_INFO: `/v3/users/dac71653e3f0429fbe0224b4d8953ec0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:05.177926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] RBAC: Authorizing `identity:delete_user(user_id=dac71653e3f0429fbe0224b4d8953ec0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:05.179874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:05.226327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2ac13f6d7d34a8884140d1f66bbf55b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.226993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2ac13f6d7d34a8884140d1f66bbf55b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.234779 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2ac13f6d7d34a8884140d1f66bbf55b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.235530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2ac13f6d7d34a8884140d1f66bbf55b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.242823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b63494b5-2530-4c50-a4f1-b0f70b5f9b63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2ac13f6d7d34a8884140d1f66bbf55b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.247148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dac71653e3f0429fbe0224b4d8953ec0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.252316 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:INSERT=4,keystone:SELECT=87,keystone:DELETE=9 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:05.255232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dac71653e3f0429fbe0224b4d8953ec0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.255905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dac71653e3f0429fbe0224b4d8953ec0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.256639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dac71653e3f0429fbe0224b4d8953ec0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.264038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-58858c99-de7a-462c-9d23-52c8f7ad9dab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dac71653e3f0429fbe0224b4d8953ec0'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.268764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1225/2461] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:57:05 2022] DELETE /identity/v3/users/f2ac13f6d7d34a8884140d1f66bbf55b => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:57:05.270028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=84,keystone:INSERT=4,keystone:DELETE=12 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:05.273255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f5d227-5378-4583-aae2-42ef58d8a8d0 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:05.273540 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f5d227-5378-4583-aae2-42ef58d8a8d0 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:05.273840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f5d227-5378-4583-aae2-42ef58d8a8d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:05.284905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1237/2462] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:57:05 2022] DELETE /identity/v3/users/dac71653e3f0429fbe0224b4d8953ec0 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:57:05.289322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a223f384-6ab7-4850-8dce-ddaac3e7a67d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:05.289674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a223f384-6ab7-4850-8dce-ddaac3e7a67d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:05.290009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a223f384-6ab7-4850-8dce-ddaac3e7a67d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:05.293241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f1f5d227-5378-4583-aae2-42ef58d8a8d0 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:05.294461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f1f5d227-5378-4583-aae2-42ef58d8a8d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:05.310039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a223f384-6ab7-4850-8dce-ddaac3e7a67d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:05.311256 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a223f384-6ab7-4850-8dce-ddaac3e7a67d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:05.352041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1226/2463] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:57:05.361944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f5d227-5378-4583-aae2-42ef58d8a8d0 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:05.363610 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f1f5d227-5378-4583-aae2-42ef58d8a8d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:05.370398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1238/2464] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:57:05.380249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a223f384-6ab7-4850-8dce-ddaac3e7a67d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:05.381613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a223f384-6ab7-4850-8dce-ddaac3e7a67d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:05.421845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f5d227-5378-4583-aae2-42ef58d8a8d0 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:05.425721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c982829-b111-45ee-8f0c-423e49714520 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:05.426355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c982829-b111-45ee-8f0c-423e49714520 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:05.426633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c982829-b111-45ee-8f0c-423e49714520 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:05.426906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c982829-b111-45ee-8f0c-423e49714520 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:05.435622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c982829-b111-45ee-8f0c-423e49714520 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:05.437175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a223f384-6ab7-4850-8dce-ddaac3e7a67d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:05.437570 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c982829-b111-45ee-8f0c-423e49714520 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:05.440702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec92c3a-3dad-485f-8664-cf09b1770417 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:05.441289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec92c3a-3dad-485f-8664-cf09b1770417 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:05.441560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec92c3a-3dad-485f-8664-cf09b1770417 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:05.441855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec92c3a-3dad-485f-8664-cf09b1770417 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:05.446868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1227/2465] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:57:05.450897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec92c3a-3dad-485f-8664-cf09b1770417 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:05.452406 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec92c3a-3dad-485f-8664-cf09b1770417 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:05.462144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1239/2466] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:57:05.824027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c982829-b111-45ee-8f0c-423e49714520 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:05.825744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2c982829-b111-45ee-8f0c-423e49714520 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:05.840925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c982829-b111-45ee-8f0c-423e49714520 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:05.853101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:05.853891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:05.854840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:05.855133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] PATH_INFO: `/v3/projects/b8122c30d932407898c08533de53ce7b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:05.857713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8122c30d932407898c08533de53ce7b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:05.860903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:05.933506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8122c30d932407898c08533de53ce7b', 'request_id': 'req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.954307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8122c30d932407898c08533de53ce7b', 'request_id': 'req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.963012 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8122c30d932407898c08533de53ce7b', 'request_id': 'req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.984990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] The token cache is being invalidate because project b8122c30d932407898c08533de53ce7b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:57:05.985317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.986041 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dd4955bc-36d1-4cf4-b3d0-034bb389fc1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:05.987552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1228/2467] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:57:05 2022] DELETE /identity/v3/projects/b8122c30d932407898c08533de53ce7b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:57:05.998342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec92c3a-3dad-485f-8664-cf09b1770417 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:05.998342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bec92c3a-3dad-485f-8664-cf09b1770417 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:06.030230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec92c3a-3dad-485f-8664-cf09b1770417 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:06.141912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:06.142737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.143022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.143377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] PATH_INFO: `/v3/projects/c55edae0711d491e9db20efce20175f5` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.144658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] RBAC: Authorizing `identity:delete_project(project_id=c55edae0711d491e9db20efce20175f5)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:06.146652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:06.185333 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:06.198806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c55edae0711d491e9db20efce20175f5', 'request_id': 'req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:06.206838 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c55edae0711d491e9db20efce20175f5', 'request_id': 'req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:06.213366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c55edae0711d491e9db20efce20175f5', 'request_id': 'req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:06.223544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] The token cache is being invalidate because project c55edae0711d491e9db20efce20175f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:57:06.223827 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:06.224448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2b0b21aa-adbf-4deb-a94d-4fbfd444762e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:06.225908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1240/2468] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:57:05 2022] DELETE /identity/v3/projects/c55edae0711d491e9db20efce20175f5 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:57:06.230861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9637f1-52e9-4346-9577-4e0b19785a53 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.231304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9637f1-52e9-4346-9577-4e0b19785a53 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.231433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9637f1-52e9-4346-9577-4e0b19785a53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.251577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ba9637f1-52e9-4346-9577-4e0b19785a53 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:06.252771 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ba9637f1-52e9-4346-9577-4e0b19785a53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:06.261048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:06.314302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1229/2469] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:06.321688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c706f7-4dc1-4220-87f5-18b0071f4ef3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.321968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c706f7-4dc1-4220-87f5-18b0071f4ef3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.322283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c706f7-4dc1-4220-87f5-18b0071f4ef3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.342151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-14c706f7-4dc1-4220-87f5-18b0071f4ef3 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:06.343363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-14c706f7-4dc1-4220-87f5-18b0071f4ef3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:06.351164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1241/2470] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:06.355199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9637f1-52e9-4346-9577-4e0b19785a53 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:06.364561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9637f1-52e9-4346-9577-4e0b19785a53 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:06.369062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f68c29-b1ff-45fb-a6ef-18625cbe741b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:06.369908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f68c29-b1ff-45fb-a6ef-18625cbe741b admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.370222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f68c29-b1ff-45fb-a6ef-18625cbe741b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.370492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f68c29-b1ff-45fb-a6ef-18625cbe741b admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.371124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f68c29-b1ff-45fb-a6ef-18625cbe741b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:57:06.371418 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f68c29-b1ff-45fb-a6ef-18625cbe741b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:06.372983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f68c29-b1ff-45fb-a6ef-18625cbe741b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:06.384587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1230/2471] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:57:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:57:06.388491 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c706f7-4dc1-4220-87f5-18b0071f4ef3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:06.397960 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c706f7-4dc1-4220-87f5-18b0071f4ef3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:06.402470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6c76c0-e03d-4d04-9d40-eb38b8a30e82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:06.403044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6c76c0-e03d-4d04-9d40-eb38b8a30e82 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.403315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6c76c0-e03d-4d04-9d40-eb38b8a30e82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.403581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6c76c0-e03d-4d04-9d40-eb38b8a30e82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.404196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6c76c0-e03d-4d04-9d40-eb38b8a30e82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:06.406087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6c76c0-e03d-4d04-9d40-eb38b8a30e82 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:06.431853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1242/2472] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:57:06 2022] POST /identity/v3/projects => generated 375 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:57:06.435833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f68c29-b1ff-45fb-a6ef-18625cbe741b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:06.445910 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f68c29-b1ff-45fb-a6ef-18625cbe741b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:06.502050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e229a092-b095-40e0-9ef4-f617b7f3148f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:06.502669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e229a092-b095-40e0-9ef4-f617b7f3148f admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.502947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e229a092-b095-40e0-9ef4-f617b7f3148f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.503225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e229a092-b095-40e0-9ef4-f617b7f3148f admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.503839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e229a092-b095-40e0-9ef4-f617b7f3148f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:06.505736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e229a092-b095-40e0-9ef4-f617b7f3148f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:06.525908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1231/2473] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:57:06 2022] POST /identity/v3/users => generated 288 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:57:06.529968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6c76c0-e03d-4d04-9d40-eb38b8a30e82 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:06.540173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6c76c0-e03d-4d04-9d40-eb38b8a30e82 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:06.544798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a820d934-7bb0-4a1d-aa67-59b5e9e7d241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:06.545409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a820d934-7bb0-4a1d-aa67-59b5e9e7d241 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.545682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a820d934-7bb0-4a1d-aa67-59b5e9e7d241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.546507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a820d934-7bb0-4a1d-aa67-59b5e9e7d241 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.547137 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a820d934-7bb0-4a1d-aa67-59b5e9e7d241 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:06.548840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a820d934-7bb0-4a1d-aa67-59b5e9e7d241 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:06.558502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1243/2474] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:57:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:57:06.562326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e229a092-b095-40e0-9ef4-f617b7f3148f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:06.572970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e229a092-b095-40e0-9ef4-f617b7f3148f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:06.581820 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e3ba16-ed37-4fde-9583-293de59894ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:06.582582 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3ba16-ed37-4fde-9583-293de59894ec admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.582960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3ba16-ed37-4fde-9583-293de59894ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.583275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3ba16-ed37-4fde-9583-293de59894ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.584054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e3ba16-ed37-4fde-9583-293de59894ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:06.585944 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e3ba16-ed37-4fde-9583-293de59894ec admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:06.599054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1232/2475] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:57:06 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:57:06.603150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a820d934-7bb0-4a1d-aa67-59b5e9e7d241 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:06.613106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a820d934-7bb0-4a1d-aa67-59b5e9e7d241 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:06.617661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21faf26-00f1-49cd-823a-d31b64c38a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:06.618335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21faf26-00f1-49cd-823a-d31b64c38a55 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.618610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21faf26-00f1-49cd-823a-d31b64c38a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.618878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21faf26-00f1-49cd-823a-d31b64c38a55 admin admin] PATH_INFO: `/v3/projects/8a3c2b18c1fb4004a482d513e018d27d/users/e50157715ad2454eb99308859b60d9f3/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.633128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21faf26-00f1-49cd-823a-d31b64c38a55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a3c2b18c1fb4004a482d513e018d27d, user_id=e50157715ad2454eb99308859b60d9f3, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:06.635766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21faf26-00f1-49cd-823a-d31b64c38a55 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:06.649042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1244/2476] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:57:06 2022] PUT /identity/v3/projects/8a3c2b18c1fb4004a482d513e018d27d/users/e50157715ad2454eb99308859b60d9f3/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:57:06.659773 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736f8b75-34b8-4930-a8dd-732a4f1138de None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.660055 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736f8b75-34b8-4930-a8dd-732a4f1138de None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.660334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736f8b75-34b8-4930-a8dd-732a4f1138de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.687347 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-736f8b75-34b8-4930-a8dd-732a4f1138de None None] MFA Rules not processed for user `e50157715ad2454eb99308859b60d9f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:06.688531 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-736f8b75-34b8-4930-a8dd-732a4f1138de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:06.751511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1233/2477] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:06.762223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21faf26-00f1-49cd-823a-d31b64c38a55 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:06.763892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c21faf26-00f1-49cd-823a-d31b64c38a55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:06.820086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21faf26-00f1-49cd-823a-d31b64c38a55 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:06.823777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f9a90-6324-41d3-ab3b-9dad6bb1f009 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:06.824313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f9a90-6324-41d3-ab3b-9dad6bb1f009 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:06.824590 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f9a90-6324-41d3-ab3b-9dad6bb1f009 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:06.824860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f9a90-6324-41d3-ab3b-9dad6bb1f009 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:06.833607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f9a90-6324-41d3-ab3b-9dad6bb1f009 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:06.835052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f9a90-6324-41d3-ab3b-9dad6bb1f009 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:06.844218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1245/2478] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:06 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:57:11.502456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1696d671-2db7-421c-8e9f-3ea0afd25e3d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:11.502958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1696d671-2db7-421c-8e9f-3ea0afd25e3d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:11.503042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1696d671-2db7-421c-8e9f-3ea0afd25e3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:11.522363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-1696d671-2db7-421c-8e9f-3ea0afd25e3d None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:11.523564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1696d671-2db7-421c-8e9f-3ea0afd25e3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:11.527780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:11.579141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1234/2479] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 00:57:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:11.628535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f9a90-6324-41d3-ab3b-9dad6bb1f009 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:11.629958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-972f9a90-6324-41d3-ab3b-9dad6bb1f009 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:11.695135 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:11.700893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f9a90-6324-41d3-ab3b-9dad6bb1f009 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:11.707810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b4591a-bc0d-44a1-9ade-890082bdfefb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:11.709128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b4591a-bc0d-44a1-9ade-890082bdfefb service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:11.709593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b4591a-bc0d-44a1-9ade-890082bdfefb service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:11.709921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b4591a-bc0d-44a1-9ade-890082bdfefb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:11.719054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b4591a-bc0d-44a1-9ade-890082bdfefb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:11.720622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b4591a-bc0d-44a1-9ade-890082bdfefb service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:11.729543 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1246/2480] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 00:57:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 00:57:17.731220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1696d671-2db7-421c-8e9f-3ea0afd25e3d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:17.845233 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1696d671-2db7-421c-8e9f-3ea0afd25e3d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:17.854222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:17.855220 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:17.855687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:17.856234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] PATH_INFO: `/v3/users/e50157715ad2454eb99308859b60d9f3` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:17.858985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e50157715ad2454eb99308859b60d9f3)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:17.861819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:17.986026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e50157715ad2454eb99308859b60d9f3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:17.987428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e50157715ad2454eb99308859b60d9f3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:17.996256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e50157715ad2454eb99308859b60d9f3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:17.997441 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e50157715ad2454eb99308859b60d9f3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:18.007171 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e50157715ad2454eb99308859b60d9f3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:18.090331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1235/2481] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:57:17 2022] DELETE /identity/v3/users/e50157715ad2454eb99308859b60d9f3 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:57:18.090483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d602b964-fa99-4e23-999f-c14a5aa0ca74 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:18.090483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d602b964-fa99-4e23-999f-c14a5aa0ca74 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:18.090483 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d602b964-fa99-4e23-999f-c14a5aa0ca74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:18.128056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-d602b964-fa99-4e23-999f-c14a5aa0ca74 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:18.133672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d602b964-fa99-4e23-999f-c14a5aa0ca74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:18.245642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1247/2482] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:18.255378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:18.256865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:18.307344 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:18.343855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d602b964-fa99-4e23-999f-c14a5aa0ca74 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:18.350194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e7a3ab-1f67-46ae-9367-8df1cd97c196 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:18.354656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062b7d08-cdb6-46f6-a7de-4c75b9c24866 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:18.355351 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b7d08-cdb6-46f6-a7de-4c75b9c24866 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:18.355700 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b7d08-cdb6-46f6-a7de-4c75b9c24866 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:18.356042 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b7d08-cdb6-46f6-a7de-4c75b9c24866 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:18.357473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d602b964-fa99-4e23-999f-c14a5aa0ca74 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:18.363634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23e530f-dbad-4527-97cc-9c5a1b1447bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:18.364324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e530f-dbad-4527-97cc-9c5a1b1447bd service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:18.365196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e530f-dbad-4527-97cc-9c5a1b1447bd service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:18.365911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e530f-dbad-4527-97cc-9c5a1b1447bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:18.368989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e23e530f-dbad-4527-97cc-9c5a1b1447bd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:18.370939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062b7d08-cdb6-46f6-a7de-4c75b9c24866 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:18.372594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062b7d08-cdb6-46f6-a7de-4c75b9c24866 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:18.385146 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1236/2483] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:57:18.481107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23e530f-dbad-4527-97cc-9c5a1b1447bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:18.485060 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23e530f-dbad-4527-97cc-9c5a1b1447bd service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:18.498889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1248/2484] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:18 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:57:18.828383 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062b7d08-cdb6-46f6-a7de-4c75b9c24866 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:18.829929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-062b7d08-cdb6-46f6-a7de-4c75b9c24866 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:18.863789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062b7d08-cdb6-46f6-a7de-4c75b9c24866 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:18.868651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:18.869581 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:18.870421 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:18.870713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] PATH_INFO: `/v3/projects/8a3c2b18c1fb4004a482d513e018d27d` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:18.872186 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] RBAC: Authorizing `identity:delete_project(project_id=8a3c2b18c1fb4004a482d513e018d27d)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:18.875065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:18.948396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8a3c2b18c1fb4004a482d513e018d27d', 'request_id': 'req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:18.958744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8a3c2b18c1fb4004a482d513e018d27d', 'request_id': 'req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:18.968180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8a3c2b18c1fb4004a482d513e018d27d', 'request_id': 'req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:18.982542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] The token cache is being invalidate because project 8a3c2b18c1fb4004a482d513e018d27d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:57:18.983213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:18.985759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3fcfe2f0-5d7f-4a55-86a2-e8102534112a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:18.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1237/2485] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:57:18 2022] DELETE /identity/v3/projects/8a3c2b18c1fb4004a482d513e018d27d => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:57:18.994621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad352971-7817-426f-b0c2-074b7a0a8b1b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:18.994812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad352971-7817-426f-b0c2-074b7a0a8b1b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:18.995170 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad352971-7817-426f-b0c2-074b7a0a8b1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:19.018887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ad352971-7817-426f-b0c2-074b7a0a8b1b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:19.020018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ad352971-7817-426f-b0c2-074b7a0a8b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:19.100255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1249/2486] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:57:19.108864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4e78fd-6808-4e41-8d40-102362b8320f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:19.109157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4e78fd-6808-4e41-8d40-102362b8320f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:19.109410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4e78fd-6808-4e41-8d40-102362b8320f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:19.136291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ea4e78fd-6808-4e41-8d40-102362b8320f None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:19.138345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ea4e78fd-6808-4e41-8d40-102362b8320f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:19.151737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1238/2487] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:19.153082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad352971-7817-426f-b0c2-074b7a0a8b1b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:19.166243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad352971-7817-426f-b0c2-074b7a0a8b1b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:19.169574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ed84cc-41fd-49a4-8ce6-5c150db164fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:19.170353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ed84cc-41fd-49a4-8ce6-5c150db164fe admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:19.170864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ed84cc-41fd-49a4-8ce6-5c150db164fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:19.171214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ed84cc-41fd-49a4-8ce6-5c150db164fe admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:19.171900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ed84cc-41fd-49a4-8ce6-5c150db164fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:57:19.172409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ed84cc-41fd-49a4-8ce6-5c150db164fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:19.174247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ed84cc-41fd-49a4-8ce6-5c150db164fe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:19.192004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1250/2488] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:57:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:57:19.195952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4e78fd-6808-4e41-8d40-102362b8320f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:19.206636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4e78fd-6808-4e41-8d40-102362b8320f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:19.214986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a303d9ec-0b8a-442d-856c-bafbef85b6c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:19.215757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a303d9ec-0b8a-442d-856c-bafbef85b6c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:19.216849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a303d9ec-0b8a-442d-856c-bafbef85b6c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:19.217170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a303d9ec-0b8a-442d-856c-bafbef85b6c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:19.218011 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a303d9ec-0b8a-442d-856c-bafbef85b6c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:19.220319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a303d9ec-0b8a-442d-856c-bafbef85b6c4 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:19.261839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1239/2489] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:57:19 2022] POST /identity/v3/projects => generated 375 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:57:19.262773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ed84cc-41fd-49a4-8ce6-5c150db164fe admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:19.278984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ed84cc-41fd-49a4-8ce6-5c150db164fe admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:19.293407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=100 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:19.352986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94c8b8a-254f-41e7-b157-ae4a9dae5aca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:19.353925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94c8b8a-254f-41e7-b157-ae4a9dae5aca admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:19.354459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94c8b8a-254f-41e7-b157-ae4a9dae5aca admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:19.354936 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94c8b8a-254f-41e7-b157-ae4a9dae5aca admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:19.355630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94c8b8a-254f-41e7-b157-ae4a9dae5aca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:19.357514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94c8b8a-254f-41e7-b157-ae4a9dae5aca admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:19.387228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a303d9ec-0b8a-442d-856c-bafbef85b6c4 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:19.393642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1251/2490] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:57:19 2022] POST /identity/v3/users => generated 288 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:57:19.400845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a303d9ec-0b8a-442d-856c-bafbef85b6c4 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:19.406933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4875f3c-a2a2-4e89-88cf-d6ee0cb6b9a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:19.406933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4875f3c-a2a2-4e89-88cf-d6ee0cb6b9a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:19.407290 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4875f3c-a2a2-4e89-88cf-d6ee0cb6b9a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:19.407557 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4875f3c-a2a2-4e89-88cf-d6ee0cb6b9a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:19.408133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4875f3c-a2a2-4e89-88cf-d6ee0cb6b9a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:19.409955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4875f3c-a2a2-4e89-88cf-d6ee0cb6b9a1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:19.438084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94c8b8a-254f-41e7-b157-ae4a9dae5aca admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:19.438330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1240/2491] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:57:19 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:57:19.438736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94c8b8a-254f-41e7-b157-ae4a9dae5aca admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:19.461389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd4899e-ff95-404e-ac75-319a11eee118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:19.462017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd4899e-ff95-404e-ac75-319a11eee118 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:19.462321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd4899e-ff95-404e-ac75-319a11eee118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:19.462592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd4899e-ff95-404e-ac75-319a11eee118 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:19.463236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd4899e-ff95-404e-ac75-319a11eee118 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:19.465078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd4899e-ff95-404e-ac75-319a11eee118 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:19.489122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1252/2492] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:57:19 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 00:57:19.493508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4875f3c-a2a2-4e89-88cf-d6ee0cb6b9a1 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:19.508001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4875f3c-a2a2-4e89-88cf-d6ee0cb6b9a1 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:19.525932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11701546-d305-4b1f-a318-55fa3d61490d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:19.526689 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11701546-d305-4b1f-a318-55fa3d61490d admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:19.527060 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11701546-d305-4b1f-a318-55fa3d61490d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:19.527241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11701546-d305-4b1f-a318-55fa3d61490d admin admin] PATH_INFO: `/v3/projects/b9e459b5a81b45e8aebe4c2409f277d0/users/24cf2d48f18841128e89bf3bea9bbf5c/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:19.545663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11701546-d305-4b1f-a318-55fa3d61490d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9e459b5a81b45e8aebe4c2409f277d0, user_id=24cf2d48f18841128e89bf3bea9bbf5c, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:19.548408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11701546-d305-4b1f-a318-55fa3d61490d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:19.566997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1241/2493] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:57:19 2022] PUT /identity/v3/projects/b9e459b5a81b45e8aebe4c2409f277d0/users/24cf2d48f18841128e89bf3bea9bbf5c/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:57:19.579433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33372db-77d8-42eb-a5fc-e606117df315 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:19.579433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33372db-77d8-42eb-a5fc-e606117df315 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:19.579433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33372db-77d8-42eb-a5fc-e606117df315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:19.622551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e33372db-77d8-42eb-a5fc-e606117df315 None None] MFA Rules not processed for user `24cf2d48f18841128e89bf3bea9bbf5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:19.624581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e33372db-77d8-42eb-a5fc-e606117df315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:19.724327 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1253/2494] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:19 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:19.742400 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11701546-d305-4b1f-a318-55fa3d61490d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:19.745401 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-11701546-d305-4b1f-a318-55fa3d61490d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:19.880141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11701546-d305-4b1f-a318-55fa3d61490d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:19.887625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6442a0e0-0327-43ea-b744-2b9b4abd856e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:19.888278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6442a0e0-0327-43ea-b744-2b9b4abd856e service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:19.889150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6442a0e0-0327-43ea-b744-2b9b4abd856e service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:19.889426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6442a0e0-0327-43ea-b744-2b9b4abd856e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:19.907166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6442a0e0-0327-43ea-b744-2b9b4abd856e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:19.910735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6442a0e0-0327-43ea-b744-2b9b4abd856e service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:19.915947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:19.928704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1242/2495] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:19 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:57:21.546836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd65e1c-cf67-4b90-b729-d0f08b66ee61 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:21.547294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd65e1c-cf67-4b90-b729-d0f08b66ee61 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:21.547393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd65e1c-cf67-4b90-b729-d0f08b66ee61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:21.568729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fbd65e1c-cf67-4b90-b729-d0f08b66ee61 None None] MFA Rules not processed for user `24cf2d48f18841128e89bf3bea9bbf5c`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:21.570512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fbd65e1c-cf67-4b90-b729-d0f08b66ee61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:21.585511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1254/2496] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:29.926205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba198f-1b74-4d67-a3e0-f7375e1430c1 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:29.927332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:29.927859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba198f-1b74-4d67-a3e0-f7375e1430c1 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:29.928465 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba198f-1b74-4d67-a3e0-f7375e1430c1 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:29.930869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1243/2497] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:57:29 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:57:29.938528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e002410-566d-4643-8fef-ed27f63ff4e8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:29.939728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e002410-566d-4643-8fef-ed27f63ff4e8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:29.940007 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e002410-566d-4643-8fef-ed27f63ff4e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:29.961063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5e002410-566d-4643-8fef-ed27f63ff4e8 None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:29.962491 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5e002410-566d-4643-8fef-ed27f63ff4e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:30.035942 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1255/2498] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:57:29 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:30.049062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ba198f-1b74-4d67-a3e0-f7375e1430c1 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:30.050916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-02ba198f-1b74-4d67-a3e0-f7375e1430c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:30.111176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ba198f-1b74-4d67-a3e0-f7375e1430c1 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:30.114827 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0888d7b-a0e3-4759-b68d-005772763a95 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc59f80f544cef8b3869e8d37bb341', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:30.115464 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0888d7b-a0e3-4759-b68d-005772763a95 service glance] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:30.115745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0888d7b-a0e3-4759-b68d-005772763a95 service glance] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:30.116018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0888d7b-a0e3-4759-b68d-005772763a95 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:30.127752 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0888d7b-a0e3-4759-b68d-005772763a95 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:30.129288 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0888d7b-a0e3-4759-b68d-005772763a95 service glance] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:30.141417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1244/2499] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:57:30.217369 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2b9a7c-d651-4a3b-bc25-69e215359acd None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:30.217800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2b9a7c-d651-4a3b-bc25-69e215359acd None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:30.218181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2b9a7c-d651-4a3b-bc25-69e215359acd None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:30.219576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1256/2500] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:57:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:57:30.224341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffc7ad2-0ac4-4bfb-ba09-7c425e724dd3 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:30.224619 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffc7ad2-0ac4-4bfb-ba09-7c425e724dd3 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:30.224918 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffc7ad2-0ac4-4bfb-ba09-7c425e724dd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:30.244406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-0ffc7ad2-0ac4-4bfb-ba09-7c425e724dd3 None None] MFA Rules not processed for user `e75811ee9ed7490985fed336ab9feb19`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:30.245604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0ffc7ad2-0ac4-4bfb-ba09-7c425e724dd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:30.298573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1245/2501] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:57:30 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:30.310474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2b9a7c-d651-4a3b-bc25-69e215359acd None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:30.312214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6f2b9a7c-d651-4a3b-bc25-69e215359acd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:30.320853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:30.325920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2b9a7c-d651-4a3b-bc25-69e215359acd None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:30.329837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-536bd996-7c8e-40a3-aabf-b122af9f31e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:30.330469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536bd996-7c8e-40a3-aabf-b122af9f31e0 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:30.330748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536bd996-7c8e-40a3-aabf-b122af9f31e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:30.331025 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-536bd996-7c8e-40a3-aabf-b122af9f31e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:30.338139 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536bd996-7c8e-40a3-aabf-b122af9f31e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:30.339551 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-536bd996-7c8e-40a3-aabf-b122af9f31e0 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:30.347657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1257/2502] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:30 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:57:30.430107 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34433aaa-8c95-4d0a-ad01-e6dad21eeb3c None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:30.430396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34433aaa-8c95-4d0a-ad01-e6dad21eeb3c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:30.430669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34433aaa-8c95-4d0a-ad01-e6dad21eeb3c None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:30.432017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1246/2503] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:57:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:57:30.436812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26866f5-0255-4047-b0da-526320370add None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:30.437106 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26866f5-0255-4047-b0da-526320370add None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:30.437382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26866f5-0255-4047-b0da-526320370add None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:30.457270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a26866f5-0255-4047-b0da-526320370add None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:30.458568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a26866f5-0255-4047-b0da-526320370add None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:30.468250 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1258/2504] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:57:30 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:57:30.479045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34433aaa-8c95-4d0a-ad01-e6dad21eeb3c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:30.490018 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34433aaa-8c95-4d0a-ad01-e6dad21eeb3c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:30.493612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e330ec-d394-4110-9cc3-257e8f75912c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:30.494496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e330ec-d394-4110-9cc3-257e8f75912c service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:30.494775 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e330ec-d394-4110-9cc3-257e8f75912c service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:30.495049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e330ec-d394-4110-9cc3-257e8f75912c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:30.506283 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e330ec-d394-4110-9cc3-257e8f75912c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:30.507703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e330ec-d394-4110-9cc3-257e8f75912c service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:30.519004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1247/2505] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:57:40.452197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:40.514732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=55 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:41.975592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55f81b2-671d-4eab-9c12-c2e2d083023f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:41.976058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55f81b2-671d-4eab-9c12-c2e2d083023f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:41.976174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55f81b2-671d-4eab-9c12-c2e2d083023f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:41.995423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f55f81b2-671d-4eab-9c12-c2e2d083023f None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:41.996607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f55f81b2-671d-4eab-9c12-c2e2d083023f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:42.054373 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1259/2506] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 00:57:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:57:42.066258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e330ec-d394-4110-9cc3-257e8f75912c service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:42.067660 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f8e330ec-d394-4110-9cc3-257e8f75912c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:42.135238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e330ec-d394-4110-9cc3-257e8f75912c service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:42.140691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e93399-73fc-4dd5-8378-5e8bd4c8cdb1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:42.141312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e93399-73fc-4dd5-8378-5e8bd4c8cdb1 service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:42.141590 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e93399-73fc-4dd5-8378-5e8bd4c8cdb1 service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:42.141864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e93399-73fc-4dd5-8378-5e8bd4c8cdb1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:42.149604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e93399-73fc-4dd5-8378-5e8bd4c8cdb1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:42.151211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e93399-73fc-4dd5-8378-5e8bd4c8cdb1 service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:42.158958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1248/2507] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 00:57:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 00:57:42.778332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3620533-16de-423e-abc1-af0a004ca8e0 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:42.779295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3620533-16de-423e-abc1-af0a004ca8e0 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:42.779636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3620533-16de-423e-abc1-af0a004ca8e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:42.799815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f3620533-16de-423e-abc1-af0a004ca8e0 None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:42.801042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f3620533-16de-423e-abc1-af0a004ca8e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:42.810233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1260/2508] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 00:57:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:42.820913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e93399-73fc-4dd5-8378-5e8bd4c8cdb1 service swift] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:42.832547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e93399-73fc-4dd5-8378-5e8bd4c8cdb1 service swift] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:42.837957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676fd39a-3ccf-4f98-b6aa-35b22b8a1805 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:42.838607 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676fd39a-3ccf-4f98-b6aa-35b22b8a1805 service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:42.839024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676fd39a-3ccf-4f98-b6aa-35b22b8a1805 service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:42.839326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676fd39a-3ccf-4f98-b6aa-35b22b8a1805 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:42.849045 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676fd39a-3ccf-4f98-b6aa-35b22b8a1805 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:42.850571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676fd39a-3ccf-4f98-b6aa-35b22b8a1805 service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:42.858685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1249/2509] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 00:57:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 00:57:43.408393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca0fc0-c6ff-46bd-9c42-abfa2f0b0359 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:43.409064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca0fc0-c6ff-46bd-9c42-abfa2f0b0359 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:43.409064 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca0fc0-c6ff-46bd-9c42-abfa2f0b0359 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:43.435917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-77ca0fc0-c6ff-46bd-9c42-abfa2f0b0359 None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:43.437528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-77ca0fc0-c6ff-46bd-9c42-abfa2f0b0359 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:43.445599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1261/2510] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 2 00:57:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:57:43.458270 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676fd39a-3ccf-4f98-b6aa-35b22b8a1805 service swift] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:43.468783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676fd39a-3ccf-4f98-b6aa-35b22b8a1805 service swift] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:43.473756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c241d6-edc5-4c18-b452-bf92a3dc18c4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:43.474395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c241d6-edc5-4c18-b452-bf92a3dc18c4 service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:43.474670 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c241d6-edc5-4c18-b452-bf92a3dc18c4 service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:43.474995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c241d6-edc5-4c18-b452-bf92a3dc18c4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:43.484980 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c241d6-edc5-4c18-b452-bf92a3dc18c4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:43.486614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c241d6-edc5-4c18-b452-bf92a3dc18c4 service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:43.495612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1250/2511] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 00:57:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 00:57:43.774689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a83ff-3652-4b25-b07b-b2ad02079ad2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:43.774968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a83ff-3652-4b25-b07b-b2ad02079ad2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:43.775237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a83ff-3652-4b25-b07b-b2ad02079ad2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:43.794428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f86a83ff-3652-4b25-b07b-b2ad02079ad2 None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:43.795633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f86a83ff-3652-4b25-b07b-b2ad02079ad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:43.805744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1262/2512] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 00:57:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:57:43.822343 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c241d6-edc5-4c18-b452-bf92a3dc18c4 service swift] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:43.840141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c241d6-edc5-4c18-b452-bf92a3dc18c4 service swift] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:43.845822 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a58b2de-9dc7-4e04-96a9-ed37868d62ed service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:43.846053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a58b2de-9dc7-4e04-96a9-ed37868d62ed service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:43.846357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a58b2de-9dc7-4e04-96a9-ed37868d62ed service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:43.846628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a58b2de-9dc7-4e04-96a9-ed37868d62ed service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:43.855342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a58b2de-9dc7-4e04-96a9-ed37868d62ed service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:43.856942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a58b2de-9dc7-4e04-96a9-ed37868d62ed service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:43.866276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1251/2513] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 00:57:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 00:57:44.270509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df66eb07-f5eb-469d-95f3-bd08a2f99ebb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:44.270860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df66eb07-f5eb-469d-95f3-bd08a2f99ebb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:44.271196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df66eb07-f5eb-469d-95f3-bd08a2f99ebb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:44.299436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-df66eb07-f5eb-469d-95f3-bd08a2f99ebb None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:44.300951 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-df66eb07-f5eb-469d-95f3-bd08a2f99ebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:44.310734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1263/2514] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 2 00:57:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:57:44.330691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a58b2de-9dc7-4e04-96a9-ed37868d62ed service swift] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:44.342640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a58b2de-9dc7-4e04-96a9-ed37868d62ed service swift] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:44.348379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac61f325-f9a1-4f5e-951f-ebfb644b2f29 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:44.349361 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac61f325-f9a1-4f5e-951f-ebfb644b2f29 service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:44.349737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac61f325-f9a1-4f5e-951f-ebfb644b2f29 service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:44.350140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac61f325-f9a1-4f5e-951f-ebfb644b2f29 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:44.364477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac61f325-f9a1-4f5e-951f-ebfb644b2f29 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:44.366167 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac61f325-f9a1-4f5e-951f-ebfb644b2f29 service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:44.377110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1252/2515] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 00:57:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 00:57:46.740570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df66eb07-f5eb-469d-95f3-bd08a2f99ebb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:46.831435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df66eb07-f5eb-469d-95f3-bd08a2f99ebb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:46.836409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:46.837040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:46.837316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:46.837587 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] PATH_INFO: `/v3/users/24cf2d48f18841128e89bf3bea9bbf5c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:46.838752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] RBAC: Authorizing `identity:delete_user(user_id=24cf2d48f18841128e89bf3bea9bbf5c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:46.841230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:46.901973 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24cf2d48f18841128e89bf3bea9bbf5c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:46.909528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24cf2d48f18841128e89bf3bea9bbf5c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:46.910249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24cf2d48f18841128e89bf3bea9bbf5c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:46.910857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24cf2d48f18841128e89bf3bea9bbf5c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:46.918433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24cf2d48f18841128e89bf3bea9bbf5c'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:46.931653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:46.938008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1264/2516] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:57:46 2022] DELETE /identity/v3/users/24cf2d48f18841128e89bf3bea9bbf5c => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:57:46.942431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed42f17e-fe3f-406b-961d-02888d69f768 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:46.942967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed42f17e-fe3f-406b-961d-02888d69f768 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:46.943056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed42f17e-fe3f-406b-961d-02888d69f768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:46.963357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ed42f17e-fe3f-406b-961d-02888d69f768 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:57:46.964620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ed42f17e-fe3f-406b-961d-02888d69f768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:47.035485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1253/2517] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:57:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 00:57:47.045725 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:47.047259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:47.108745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-044120b4-076a-4d62-ac2b-f1c06f461f8d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:47.113440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787b011d-68ba-4216-b4cb-30c9e3e8bea5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:47.114203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787b011d-68ba-4216-b4cb-30c9e3e8bea5 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:47.114578 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787b011d-68ba-4216-b4cb-30c9e3e8bea5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:47.114952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787b011d-68ba-4216-b4cb-30c9e3e8bea5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:47.126487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787b011d-68ba-4216-b4cb-30c9e3e8bea5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:47.128174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787b011d-68ba-4216-b4cb-30c9e3e8bea5 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:47.146768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1265/2518] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:57:47.398731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed42f17e-fe3f-406b-961d-02888d69f768 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:47.400420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ed42f17e-fe3f-406b-961d-02888d69f768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:47.414008 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed42f17e-fe3f-406b-961d-02888d69f768 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:47.419612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:47.420393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:47.420796 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:47.421067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] PATH_INFO: `/v3/projects/b9e459b5a81b45e8aebe4c2409f277d0` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:47.422505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9e459b5a81b45e8aebe4c2409f277d0)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:47.424615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:47.481087 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:57:47.494340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9e459b5a81b45e8aebe4c2409f277d0', 'request_id': 'req-b07aba08-b64f-4c77-a51d-4d74f786362d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:47.502160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9e459b5a81b45e8aebe4c2409f277d0', 'request_id': 'req-b07aba08-b64f-4c77-a51d-4d74f786362d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:47.508535 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9e459b5a81b45e8aebe4c2409f277d0', 'request_id': 'req-b07aba08-b64f-4c77-a51d-4d74f786362d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:47.519413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] The token cache is being invalidate because project b9e459b5a81b45e8aebe4c2409f277d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:57:47.519698 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:47.520357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-b07aba08-b64f-4c77-a51d-4d74f786362d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:57:47.521897 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1254/2519] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:57:47 2022] DELETE /identity/v3/projects/b9e459b5a81b45e8aebe4c2409f277d0 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:57:52.628786 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787b011d-68ba-4216-b4cb-30c9e3e8bea5 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:57:52.630233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-787b011d-68ba-4216-b4cb-30c9e3e8bea5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:52.703297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787b011d-68ba-4216-b4cb-30c9e3e8bea5 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:57:52.766545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fb724b-be83-452d-b7e0-7b655853e5fa service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:57:52.767148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fb724b-be83-452d-b7e0-7b655853e5fa service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:57:52.767419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fb724b-be83-452d-b7e0-7b655853e5fa service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:57:52.767684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fb724b-be83-452d-b7e0-7b655853e5fa service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:57:52.769392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a5fb724b-be83-452d-b7e0-7b655853e5fa service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:57:52.780767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fb724b-be83-452d-b7e0-7b655853e5fa service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:57:52.782347 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fb724b-be83-452d-b7e0-7b655853e5fa service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:57:52.792033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1266/2520] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:57:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:57:57.516095 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=12,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:58:02.787812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=85 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:58:04.858195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aae6258-7758-47b8-b7bf-d3e3a63eaf2d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:58:04.858655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aae6258-7758-47b8-b7bf-d3e3a63eaf2d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:58:04.858760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aae6258-7758-47b8-b7bf-d3e3a63eaf2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:58:04.878553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-7aae6258-7758-47b8-b7bf-d3e3a63eaf2d None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:58:04.879726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7aae6258-7758-47b8-b7bf-d3e3a63eaf2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:58:04.932290 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1255/2521] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 2 00:58:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:58:04.944928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fb724b-be83-452d-b7e0-7b655853e5fa service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:58:04.946359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a5fb724b-be83-452d-b7e0-7b655853e5fa service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:58:05.012964 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fb724b-be83-452d-b7e0-7b655853e5fa service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:58:05.017948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d034a971-5274-4cf3-a0cd-ce36d99d90cf service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:58:05.018589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d034a971-5274-4cf3-a0cd-ce36d99d90cf service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:58:05.018866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d034a971-5274-4cf3-a0cd-ce36d99d90cf service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:58:05.019253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d034a971-5274-4cf3-a0cd-ce36d99d90cf service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:58:05.035104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d034a971-5274-4cf3-a0cd-ce36d99d90cf service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:58:05.036746 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d034a971-5274-4cf3-a0cd-ce36d99d90cf service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:58:05.045969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1267/2522] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 00:58:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 00:58:14.925712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=25 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:58:15.042146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=32 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:58:27.599081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79441f82-0855-4b59-a80d-04c808402777 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:58:27.599609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79441f82-0855-4b59-a80d-04c808402777 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:58:27.599609 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79441f82-0855-4b59-a80d-04c808402777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:58:27.617906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-79441f82-0855-4b59-a80d-04c808402777 None None] MFA Rules not processed for user `afe98894d83f4678bb134782d90706e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:58:27.619380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-79441f82-0855-4b59-a80d-04c808402777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:58:27.673293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1256/2523] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:58:27 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:58:37.667284 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=25 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:58:53.075809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb71bf96-db58-4624-917a-56ea442c09d3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:58:53.076324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb71bf96-db58-4624-917a-56ea442c09d3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:58:53.076324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb71bf96-db58-4624-917a-56ea442c09d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:58:53.095025 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fb71bf96-db58-4624-917a-56ea442c09d3 None None] MFA Rules not processed for user `a8894fd1044241ee8ba84927b6e8326a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:58:53.096243 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fb71bf96-db58-4624-917a-56ea442c09d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:58:53.158527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1268/2524] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:58:53 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:03.152118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=25 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:30.365422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79441f82-0855-4b59-a80d-04c808402777 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:30.367423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-79441f82-0855-4b59-a80d-04c808402777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:30.435022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79441f82-0855-4b59-a80d-04c808402777 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:30.439365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:30.439981 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:30.440255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:30.440525 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] PATH_INFO: `/v3/users/afe98894d83f4678bb134782d90706e8` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:30.441664 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] RBAC: Authorizing `identity:delete_user(user_id=afe98894d83f4678bb134782d90706e8)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:30.443603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:30.487878 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'afe98894d83f4678bb134782d90706e8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:30.488463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'afe98894d83f4678bb134782d90706e8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:30.495613 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'afe98894d83f4678bb134782d90706e8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:30.496231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'afe98894d83f4678bb134782d90706e8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:30.503155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-f3e8693b-86d9-44d4-8390-f0dfdd4fce01 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'afe98894d83f4678bb134782d90706e8'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:30.523915 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1257/2525] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:59:30 2022] DELETE /identity/v3/users/afe98894d83f4678bb134782d90706e8 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:30.686920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb71bf96-db58-4624-917a-56ea442c09d3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:30.702489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb71bf96-db58-4624-917a-56ea442c09d3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:30.773837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:30.774661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:30.775166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:30.776121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] PATH_INFO: `/v3/projects/4c22ffb83d7d41828375606fcc6da53f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:30.777536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c22ffb83d7d41828375606fcc6da53f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:30.779693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:30.833181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c22ffb83d7d41828375606fcc6da53f', 'request_id': 'req-f8a294ae-7534-4444-9485-c53767e007ed'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:30.840420 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c22ffb83d7d41828375606fcc6da53f', 'request_id': 'req-f8a294ae-7534-4444-9485-c53767e007ed'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:30.847380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c22ffb83d7d41828375606fcc6da53f', 'request_id': 'req-f8a294ae-7534-4444-9485-c53767e007ed'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:30.859570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] The token cache is being invalidate because project 4c22ffb83d7d41828375606fcc6da53f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:59:30.859861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:30.860520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-f8a294ae-7534-4444-9485-c53767e007ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:30.862205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1269/2526] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:59:30 2022] DELETE /identity/v3/projects/4c22ffb83d7d41828375606fcc6da53f => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:30.867807 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bf0b86-b355-4b15-b328-5648d1ca2d21 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:30.868196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bf0b86-b355-4b15-b328-5648d1ca2d21 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:30.868477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bf0b86-b355-4b15-b328-5648d1ca2d21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:30.889236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-99bf0b86-b355-4b15-b328-5648d1ca2d21 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:30.890471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-99bf0b86-b355-4b15-b328-5648d1ca2d21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:30.948809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1258/2527] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:30.956205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aac5a05-c3e8-465a-93c2-c3439a6588d9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:30.956503 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aac5a05-c3e8-465a-93c2-c3439a6588d9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:30.956773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aac5a05-c3e8-465a-93c2-c3439a6588d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:30.977280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3aac5a05-c3e8-465a-93c2-c3439a6588d9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:30.978529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3aac5a05-c3e8-465a-93c2-c3439a6588d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:30.986938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1270/2528] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:30.991164 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bf0b86-b355-4b15-b328-5648d1ca2d21 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:31.001184 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bf0b86-b355-4b15-b328-5648d1ca2d21 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:31.005749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c153ae8f-3e2f-4cc8-879b-1c7398562cd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:31.006402 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c153ae8f-3e2f-4cc8-879b-1c7398562cd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:31.006682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c153ae8f-3e2f-4cc8-879b-1c7398562cd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:31.006954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c153ae8f-3e2f-4cc8-879b-1c7398562cd7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:31.007592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c153ae8f-3e2f-4cc8-879b-1c7398562cd7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:59:31.007892 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c153ae8f-3e2f-4cc8-879b-1c7398562cd7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:31.009467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c153ae8f-3e2f-4cc8-879b-1c7398562cd7 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:31.021022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1259/2529] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:59:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:59:31.025818 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aac5a05-c3e8-465a-93c2-c3439a6588d9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:31.035683 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aac5a05-c3e8-465a-93c2-c3439a6588d9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:31.040904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f23a1f1-11d5-4a77-891d-dde20de39122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:31.041498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f23a1f1-11d5-4a77-891d-dde20de39122 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:31.041771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f23a1f1-11d5-4a77-891d-dde20de39122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:31.042042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f23a1f1-11d5-4a77-891d-dde20de39122 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:31.042687 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f23a1f1-11d5-4a77-891d-dde20de39122 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:31.044573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f23a1f1-11d5-4a77-891d-dde20de39122 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:31.070489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1271/2530] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:59:31 2022] POST /identity/v3/projects => generated 365 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:31.074799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c153ae8f-3e2f-4cc8-879b-1c7398562cd7 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:31.084585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c153ae8f-3e2f-4cc8-879b-1c7398562cd7 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:31.088654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:31.142368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7a0f56-07be-447c-9764-666131cad952 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:31.142965 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7a0f56-07be-447c-9764-666131cad952 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:31.143241 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7a0f56-07be-447c-9764-666131cad952 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:31.143510 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7a0f56-07be-447c-9764-666131cad952 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:31.144133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7a0f56-07be-447c-9764-666131cad952 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:31.146014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7a0f56-07be-447c-9764-666131cad952 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:31.166580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1260/2531] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:59:31 2022] POST /identity/v3/users => generated 283 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:31.171015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f23a1f1-11d5-4a77-891d-dde20de39122 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:31.181250 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f23a1f1-11d5-4a77-891d-dde20de39122 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:31.185721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a0d26c-ae1b-40b1-a1ea-496bfd7e8c5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:31.186351 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a0d26c-ae1b-40b1-a1ea-496bfd7e8c5d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:31.186625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a0d26c-ae1b-40b1-a1ea-496bfd7e8c5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:31.186890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a0d26c-ae1b-40b1-a1ea-496bfd7e8c5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:31.187509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a0d26c-ae1b-40b1-a1ea-496bfd7e8c5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:31.189370 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a0d26c-ae1b-40b1-a1ea-496bfd7e8c5d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:31.199004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1272/2532] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:31 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:31.202566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7a0f56-07be-447c-9764-666131cad952 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:31.212798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7a0f56-07be-447c-9764-666131cad952 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:31.217300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d915f6f5-51e4-44fb-a29f-b1939fa0eb70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:31.217883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d915f6f5-51e4-44fb-a29f-b1939fa0eb70 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:31.218190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d915f6f5-51e4-44fb-a29f-b1939fa0eb70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:31.218463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d915f6f5-51e4-44fb-a29f-b1939fa0eb70 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:31.219091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d915f6f5-51e4-44fb-a29f-b1939fa0eb70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:31.220640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d915f6f5-51e4-44fb-a29f-b1939fa0eb70 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:31.231465 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1261/2533] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:31.235389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a0d26c-ae1b-40b1-a1ea-496bfd7e8c5d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:31.245349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a0d26c-ae1b-40b1-a1ea-496bfd7e8c5d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:31.249793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919988ba-9ff6-4d63-bb8b-7347969c1aec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:31.250463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919988ba-9ff6-4d63-bb8b-7347969c1aec admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:31.250736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919988ba-9ff6-4d63-bb8b-7347969c1aec admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:31.251004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919988ba-9ff6-4d63-bb8b-7347969c1aec admin admin] PATH_INFO: `/v3/projects/9c7ba9498e2645bb914a43685ab6795f/users/c24d6c5666964513a0d641eccb6b58ad/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:31.265047 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919988ba-9ff6-4d63-bb8b-7347969c1aec admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c7ba9498e2645bb914a43685ab6795f, user_id=c24d6c5666964513a0d641eccb6b58ad, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:31.267770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919988ba-9ff6-4d63-bb8b-7347969c1aec admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:31.280969 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1273/2534] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:59:31 2022] PUT /identity/v3/projects/9c7ba9498e2645bb914a43685ab6795f/users/c24d6c5666964513a0d641eccb6b58ad/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:59:31.291699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43a43a9-12c1-4c11-acf9-998890795eb2 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:31.291988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43a43a9-12c1-4c11-acf9-998890795eb2 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:31.292270 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43a43a9-12c1-4c11-acf9-998890795eb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:31.318961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-c43a43a9-12c1-4c11-acf9-998890795eb2 None None] MFA Rules not processed for user `c24d6c5666964513a0d641eccb6b58ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:31.320162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c43a43a9-12c1-4c11-acf9-998890795eb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:31.383244 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1262/2535] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:31 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:31.394041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919988ba-9ff6-4d63-bb8b-7347969c1aec admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:31.395728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-919988ba-9ff6-4d63-bb8b-7347969c1aec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:31.430265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:31.458843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919988ba-9ff6-4d63-bb8b-7347969c1aec admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:31.462975 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff03a161-0924-4078-9d6c-db0d42db09dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:31.463557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff03a161-0924-4078-9d6c-db0d42db09dd service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:31.463828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff03a161-0924-4078-9d6c-db0d42db09dd service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:31.464093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff03a161-0924-4078-9d6c-db0d42db09dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:31.472137 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff03a161-0924-4078-9d6c-db0d42db09dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:31.473556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff03a161-0924-4078-9d6c-db0d42db09dd service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:31.482115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1274/2536] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:31 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:34.236688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43a43a9-12c1-4c11-acf9-998890795eb2 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:34.238112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c43a43a9-12c1-4c11-acf9-998890795eb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:34.250854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43a43a9-12c1-4c11-acf9-998890795eb2 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:34.254519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c2ee1-76ac-43d9-904b-c26739fea5d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:34.255119 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c2ee1-76ac-43d9-904b-c26739fea5d9 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:34.255399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c2ee1-76ac-43d9-904b-c26739fea5d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:34.255702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c2ee1-76ac-43d9-904b-c26739fea5d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:34.257397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-275c2ee1-76ac-43d9-904b-c26739fea5d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:34.329259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275c2ee1-76ac-43d9-904b-c26739fea5d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:34.330782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275c2ee1-76ac-43d9-904b-c26739fea5d9 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:34.337890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:34.344108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1263/2537] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:35.737664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff03a161-0924-4078-9d6c-db0d42db09dd service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:35.746997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff03a161-0924-4078-9d6c-db0d42db09dd service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:35.750924 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e37ece-1752-4080-89a1-4f8c2dfb714b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:35.751649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e37ece-1752-4080-89a1-4f8c2dfb714b service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:35.752000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e37ece-1752-4080-89a1-4f8c2dfb714b service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:35.752183 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e37ece-1752-4080-89a1-4f8c2dfb714b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:35.754346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-92e37ece-1752-4080-89a1-4f8c2dfb714b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:35.767876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e37ece-1752-4080-89a1-4f8c2dfb714b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:35.769304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e37ece-1752-4080-89a1-4f8c2dfb714b service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:35.801768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1275/2538] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:39.725591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f40a17-7bf6-4dd6-bfb7-cbb38e37af5b None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:39.726623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f40a17-7bf6-4dd6-bfb7-cbb38e37af5b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:39.726730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f40a17-7bf6-4dd6-bfb7-cbb38e37af5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:39.745778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-d2f40a17-7bf6-4dd6-bfb7-cbb38e37af5b None None] MFA Rules not processed for user `6415bc1076204c78ad819ab2836a3652`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:39.747031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d2f40a17-7bf6-4dd6-bfb7-cbb38e37af5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:39.800263 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1264/2539] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:39 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:40.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e37ece-1752-4080-89a1-4f8c2dfb714b service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:40.248787 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e37ece-1752-4080-89a1-4f8c2dfb714b service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:40.253843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:40.254518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:40.254791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:40.255053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] PATH_INFO: `/v3/users/c24d6c5666964513a0d641eccb6b58ad` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:40.256146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c24d6c5666964513a0d641eccb6b58ad)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:40.258147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:40.306492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c24d6c5666964513a0d641eccb6b58ad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:40.313191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c24d6c5666964513a0d641eccb6b58ad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:40.313846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c24d6c5666964513a0d641eccb6b58ad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:40.314518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c24d6c5666964513a0d641eccb6b58ad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:40.321650 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c24d6c5666964513a0d641eccb6b58ad'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:40.344775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1276/2540] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:59:40 2022] DELETE /identity/v3/users/c24d6c5666964513a0d641eccb6b58ad => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:40.346647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a27521-22ed-47d9-be43-add71c4adc48 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:40.346961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a27521-22ed-47d9-be43-add71c4adc48 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:40.347261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a27521-22ed-47d9-be43-add71c4adc48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:40.369927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a1a27521-22ed-47d9-be43-add71c4adc48 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:40.371179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a1a27521-22ed-47d9-be43-add71c4adc48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:40.449341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1265/2541] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:40.457156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:40.459297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:40.516928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:40.541903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e420d1-4a84-4f1c-9995-a781362b23c6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:40.546094 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc18e8fc-ee45-444b-a2c6-47c1926daa4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:40.546676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc18e8fc-ee45-444b-a2c6-47c1926daa4c service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:40.547026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc18e8fc-ee45-444b-a2c6-47c1926daa4c service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:40.547364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc18e8fc-ee45-444b-a2c6-47c1926daa4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:40.561252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc18e8fc-ee45-444b-a2c6-47c1926daa4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:40.561252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc18e8fc-ee45-444b-a2c6-47c1926daa4c service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:40.570545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1277/2542] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:40.857792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a27521-22ed-47d9-be43-add71c4adc48 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:40.861800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a1a27521-22ed-47d9-be43-add71c4adc48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:40.880875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a27521-22ed-47d9-be43-add71c4adc48 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:40.888112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:40.888932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:40.889335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:40.889716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] PATH_INFO: `/v3/projects/9c7ba9498e2645bb914a43685ab6795f` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:40.891112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c7ba9498e2645bb914a43685ab6795f)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:40.893856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:40.950227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c7ba9498e2645bb914a43685ab6795f', 'request_id': 'req-dee81542-23c6-43b4-87bb-0057f5f537c3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:40.958586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c7ba9498e2645bb914a43685ab6795f', 'request_id': 'req-dee81542-23c6-43b4-87bb-0057f5f537c3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:40.966226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c7ba9498e2645bb914a43685ab6795f', 'request_id': 'req-dee81542-23c6-43b4-87bb-0057f5f537c3'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:40.980537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] The token cache is being invalidate because project 9c7ba9498e2645bb914a43685ab6795f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:59:40.980831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:40.981507 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-dee81542-23c6-43b4-87bb-0057f5f537c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:40.983206 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1266/2543] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:59:40 2022] DELETE /identity/v3/projects/9c7ba9498e2645bb914a43685ab6795f => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:40.988935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce22e51-ebcd-4d9c-92f9-2096afe8608d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:40.989104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce22e51-ebcd-4d9c-92f9-2096afe8608d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:40.989426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce22e51-ebcd-4d9c-92f9-2096afe8608d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.009550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-cce22e51-ebcd-4d9c-92f9-2096afe8608d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:41.010799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-cce22e51-ebcd-4d9c-92f9-2096afe8608d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:41.079942 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1278/2544] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:41.088255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed77499d-0ca2-4d32-9438-7e5378942c89 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.088537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed77499d-0ca2-4d32-9438-7e5378942c89 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.088814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed77499d-0ca2-4d32-9438-7e5378942c89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.109587 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ed77499d-0ca2-4d32-9438-7e5378942c89 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:41.110893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ed77499d-0ca2-4d32-9438-7e5378942c89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:41.119132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1267/2545] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:41.123628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce22e51-ebcd-4d9c-92f9-2096afe8608d None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.134630 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce22e51-ebcd-4d9c-92f9-2096afe8608d None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:41.139159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6a384c-36c0-4e66-9094-595eba816710 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:41.139750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6a384c-36c0-4e66-9094-595eba816710 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.140028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6a384c-36c0-4e66-9094-595eba816710 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.140326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6a384c-36c0-4e66-9094-595eba816710 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.140964 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6a384c-36c0-4e66-9094-595eba816710 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:59:41.141263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6a384c-36c0-4e66-9094-595eba816710 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:41.143043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc6a384c-36c0-4e66-9094-595eba816710 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:41.155195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1279/2546] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:59:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:59:41.159423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed77499d-0ca2-4d32-9438-7e5378942c89 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.169230 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed77499d-0ca2-4d32-9438-7e5378942c89 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:41.174172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558a8d3e-7f73-4a67-91a7-4ca6d9fb189b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:41.174776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a8d3e-7f73-4a67-91a7-4ca6d9fb189b admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.175054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a8d3e-7f73-4a67-91a7-4ca6d9fb189b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.175328 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a8d3e-7f73-4a67-91a7-4ca6d9fb189b admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.175970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558a8d3e-7f73-4a67-91a7-4ca6d9fb189b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:41.177879 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558a8d3e-7f73-4a67-91a7-4ca6d9fb189b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:41.200306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:41.208141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1268/2547] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:59:41 2022] POST /identity/v3/projects => generated 363 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:41.212519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6a384c-36c0-4e66-9094-595eba816710 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.222848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc6a384c-36c0-4e66-9094-595eba816710 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:41.281698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58366b7-4673-480f-a407-ae37e34c3992 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:41.282337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58366b7-4673-480f-a407-ae37e34c3992 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.282618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58366b7-4673-480f-a407-ae37e34c3992 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.282889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58366b7-4673-480f-a407-ae37e34c3992 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.283517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58366b7-4673-480f-a407-ae37e34c3992 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:41.285510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58366b7-4673-480f-a407-ae37e34c3992 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:41.307733 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1280/2548] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:59:41 2022] POST /identity/v3/users => generated 282 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:41.312121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558a8d3e-7f73-4a67-91a7-4ca6d9fb189b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.323705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558a8d3e-7f73-4a67-91a7-4ca6d9fb189b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:41.328982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097995e5-efcc-4e32-93ea-4de7d86ad79f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:41.329584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097995e5-efcc-4e32-93ea-4de7d86ad79f admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.329862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097995e5-efcc-4e32-93ea-4de7d86ad79f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.330162 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097995e5-efcc-4e32-93ea-4de7d86ad79f admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.330800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097995e5-efcc-4e32-93ea-4de7d86ad79f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:41.332365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097995e5-efcc-4e32-93ea-4de7d86ad79f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:41.343280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1269/2549] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:41 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:41.347619 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58366b7-4673-480f-a407-ae37e34c3992 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.357355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58366b7-4673-480f-a407-ae37e34c3992 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:41.361844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedc0702-b30b-4da3-b324-ffc076b97347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:41.362453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedc0702-b30b-4da3-b324-ffc076b97347 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.362734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedc0702-b30b-4da3-b324-ffc076b97347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.362999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedc0702-b30b-4da3-b324-ffc076b97347 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.363618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedc0702-b30b-4da3-b324-ffc076b97347 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:41.365182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cedc0702-b30b-4da3-b324-ffc076b97347 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:41.375662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1281/2550] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:41 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:41.379766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097995e5-efcc-4e32-93ea-4de7d86ad79f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.390142 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097995e5-efcc-4e32-93ea-4de7d86ad79f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:41.394667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8f67ed-137b-4afb-91a6-99d990a6b6d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:41.395333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8f67ed-137b-4afb-91a6-99d990a6b6d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.395596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8f67ed-137b-4afb-91a6-99d990a6b6d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.395875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8f67ed-137b-4afb-91a6-99d990a6b6d1 admin admin] PATH_INFO: `/v3/projects/12c87799cd9a4b36bead449c22bafa59/users/2e7628060b164b26b7af194993fab104/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.411132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8f67ed-137b-4afb-91a6-99d990a6b6d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=12c87799cd9a4b36bead449c22bafa59, user_id=2e7628060b164b26b7af194993fab104, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:41.413798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8f67ed-137b-4afb-91a6-99d990a6b6d1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:41.426155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1270/2551] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:59:41 2022] PUT /identity/v3/projects/12c87799cd9a4b36bead449c22bafa59/users/2e7628060b164b26b7af194993fab104/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:41.436458 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2109f7-f108-42b3-9f2a-a27f4181e3ac None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.436836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2109f7-f108-42b3-9f2a-a27f4181e3ac None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.437104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2109f7-f108-42b3-9f2a-a27f4181e3ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.464821 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-db2109f7-f108-42b3-9f2a-a27f4181e3ac None None] MFA Rules not processed for user `2e7628060b164b26b7af194993fab104`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:41.466016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-db2109f7-f108-42b3-9f2a-a27f4181e3ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:41.498227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:41.541968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1282/2552] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:41 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:41.546626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8f67ed-137b-4afb-91a6-99d990a6b6d1 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.636596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8f67ed-137b-4afb-91a6-99d990a6b6d1 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:41.642020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d672d36c-00b2-49c0-9155-827ba483dd89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:41.642749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d672d36c-00b2-49c0-9155-827ba483dd89 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.643138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d672d36c-00b2-49c0-9155-827ba483dd89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.643568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d672d36c-00b2-49c0-9155-827ba483dd89 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.644285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d672d36c-00b2-49c0-9155-827ba483dd89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:41.648427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d672d36c-00b2-49c0-9155-827ba483dd89 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:41.696235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1271/2553] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:59:41 2022] POST /identity/v3/projects => generated 365 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:41.697058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2109f7-f108-42b3-9f2a-a27f4181e3ac None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.708193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2109f7-f108-42b3-9f2a-a27f4181e3ac None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:41.792878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6d3e5b-d3d7-4e4e-a293-1302420b2e53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:41.793614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6d3e5b-d3d7-4e4e-a293-1302420b2e53 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.793897 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6d3e5b-d3d7-4e4e-a293-1302420b2e53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.794211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6d3e5b-d3d7-4e4e-a293-1302420b2e53 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.795117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6d3e5b-d3d7-4e4e-a293-1302420b2e53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:41.797087 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6d3e5b-d3d7-4e4e-a293-1302420b2e53 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:41.824594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1283/2554] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:59:41 2022] POST /identity/v3/users => generated 289 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 00:59:41.828840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d672d36c-00b2-49c0-9155-827ba483dd89 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.856441 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d672d36c-00b2-49c0-9155-827ba483dd89 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:41.861973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc3d49-b1c2-467b-8f02-4a2f86305042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:41.862775 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc3d49-b1c2-467b-8f02-4a2f86305042 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.863078 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc3d49-b1c2-467b-8f02-4a2f86305042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.863353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc3d49-b1c2-467b-8f02-4a2f86305042 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.864070 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc3d49-b1c2-467b-8f02-4a2f86305042 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:41.866300 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc3d49-b1c2-467b-8f02-4a2f86305042 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:41.877925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1272/2555] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:41 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:41.882462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6d3e5b-d3d7-4e4e-a293-1302420b2e53 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.892501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6d3e5b-d3d7-4e4e-a293-1302420b2e53 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:41.898513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eff8d79-9466-4ec8-8674-61bd57dfc11c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:41.899651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eff8d79-9466-4ec8-8674-61bd57dfc11c admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:41.899893 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eff8d79-9466-4ec8-8674-61bd57dfc11c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:41.900166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eff8d79-9466-4ec8-8674-61bd57dfc11c admin admin] PATH_INFO: `/v3/projects/b76f489efaaa48cf89b90001061d7741/users/3357f84452d34260b7eaeb9b8f28f9f7/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:41.919250 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eff8d79-9466-4ec8-8674-61bd57dfc11c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b76f489efaaa48cf89b90001061d7741, user_id=3357f84452d34260b7eaeb9b8f28f9f7, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:41.923198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eff8d79-9466-4ec8-8674-61bd57dfc11c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:41.936978 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1284/2556] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:59:41 2022] PUT /identity/v3/projects/b76f489efaaa48cf89b90001061d7741/users/3357f84452d34260b7eaeb9b8f28f9f7/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:41.940934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc3d49-b1c2-467b-8f02-4a2f86305042 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:41.954804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc3d49-b1c2-467b-8f02-4a2f86305042 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:42.010979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ede9c6-d1a6-49d3-9f16-b31f3b56a485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:42.011585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ede9c6-d1a6-49d3-9f16-b31f3b56a485 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:42.011864 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ede9c6-d1a6-49d3-9f16-b31f3b56a485 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:42.012136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ede9c6-d1a6-49d3-9f16-b31f3b56a485 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:42.012764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ede9c6-d1a6-49d3-9f16-b31f3b56a485 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:42.014372 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ede9c6-d1a6-49d3-9f16-b31f3b56a485 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:42.024466 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1273/2557] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:41 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:42.028665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eff8d79-9466-4ec8-8674-61bd57dfc11c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:42.038943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eff8d79-9466-4ec8-8674-61bd57dfc11c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:42.043848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768bb37a-921c-4968-be04-b8fabda2d8ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:42.044439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768bb37a-921c-4968-be04-b8fabda2d8ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:42.044710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768bb37a-921c-4968-be04-b8fabda2d8ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:42.044980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768bb37a-921c-4968-be04-b8fabda2d8ad admin admin] PATH_INFO: `/v3/projects/b76f489efaaa48cf89b90001061d7741/users/3357f84452d34260b7eaeb9b8f28f9f7/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:42.046919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768bb37a-921c-4968-be04-b8fabda2d8ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=b76f489efaaa48cf89b90001061d7741, user_id=3357f84452d34260b7eaeb9b8f28f9f7, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:42.049518 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768bb37a-921c-4968-be04-b8fabda2d8ad admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:42.059866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1285/2558] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:59:42 2022] PUT /identity/v3/projects/b76f489efaaa48cf89b90001061d7741/users/3357f84452d34260b7eaeb9b8f28f9f7/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:42.071596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8b3954-6709-48d6-8172-504b5b68132f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:42.071875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8b3954-6709-48d6-8172-504b5b68132f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:42.072145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8b3954-6709-48d6-8172-504b5b68132f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:42.099518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ef8b3954-6709-48d6-8172-504b5b68132f None None] MFA Rules not processed for user `3357f84452d34260b7eaeb9b8f28f9f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:42.100698 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ef8b3954-6709-48d6-8172-504b5b68132f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:42.123460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:42.167269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1274/2559] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:42 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:42.179031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768bb37a-921c-4968-be04-b8fabda2d8ad admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:42.180728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-768bb37a-921c-4968-be04-b8fabda2d8ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:42.236565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768bb37a-921c-4968-be04-b8fabda2d8ad admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:42.240286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1562703-7774-434c-80c0-c77d0d7392d6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:42.240870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1562703-7774-434c-80c0-c77d0d7392d6 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:42.241149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1562703-7774-434c-80c0-c77d0d7392d6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:42.242216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1562703-7774-434c-80c0-c77d0d7392d6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:42.251067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1562703-7774-434c-80c0-c77d0d7392d6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:42.252476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1562703-7774-434c-80c0-c77d0d7392d6 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:42.262221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1286/2560] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:42 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:42.336832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8b3954-6709-48d6-8172-504b5b68132f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:42.348551 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8b3954-6709-48d6-8172-504b5b68132f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:42.354951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09ad03a-2ed3-4628-be99-f9da39271086 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:42.355591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09ad03a-2ed3-4628-be99-f9da39271086 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:42.355859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09ad03a-2ed3-4628-be99-f9da39271086 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:42.356134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09ad03a-2ed3-4628-be99-f9da39271086 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:42.424178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09ad03a-2ed3-4628-be99-f9da39271086 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:42.425823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09ad03a-2ed3-4628-be99-f9da39271086 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:42.436240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1275/2561] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:42 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:43.291090 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858fb672-0042-4034-8ff5-3a5452865b63 None None] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:43.291653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858fb672-0042-4034-8ff5-3a5452865b63 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:43.291653 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858fb672-0042-4034-8ff5-3a5452865b63 None None] PATH_INFO: `/` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:43.294185 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1287/2562] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:59:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:59:43.300398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409bc39f-b02f-4919-b53c-89b6f42667e7 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:43.300883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409bc39f-b02f-4919-b53c-89b6f42667e7 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:43.300982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409bc39f-b02f-4919-b53c-89b6f42667e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:43.321626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-409bc39f-b02f-4919-b53c-89b6f42667e7 None None] MFA Rules not processed for user `e75811ee9ed7490985fed336ab9feb19`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:43.322868 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-409bc39f-b02f-4919-b53c-89b6f42667e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:43.374680 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1276/2563] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:59:43 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:43.387432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858fb672-0042-4034-8ff5-3a5452865b63 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:43.389261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-858fb672-0042-4034-8ff5-3a5452865b63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:43.403735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858fb672-0042-4034-8ff5-3a5452865b63 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:43.407353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb70deb-2fc3-4f0e-87a1-d4e3ffdc98f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:43.408043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb70deb-2fc3-4f0e-87a1-d4e3ffdc98f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:43.408392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb70deb-2fc3-4f0e-87a1-d4e3ffdc98f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:43.408668 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb70deb-2fc3-4f0e-87a1-d4e3ffdc98f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:43.416286 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb70deb-2fc3-4f0e-87a1-d4e3ffdc98f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:43.417873 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb70deb-2fc3-4f0e-87a1-d4e3ffdc98f7 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:43.425871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1288/2564] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:43 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:59:43.510808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483bc62e-9ef4-422a-ad04-7d0a01ed9010 None None] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:43.511223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483bc62e-9ef4-422a-ad04-7d0a01ed9010 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:43.511568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483bc62e-9ef4-422a-ad04-7d0a01ed9010 None None] PATH_INFO: `/` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:43.513963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1277/2565] 10.222.0.25 () {54 vars in 921 bytes} [Sun Oct 2 00:59:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 02 00:59:43.520173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ff6f95-83ea-41c0-aa8b-929c09577aee None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:43.520459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ff6f95-83ea-41c0-aa8b-929c09577aee None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:43.520737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ff6f95-83ea-41c0-aa8b-929c09577aee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:43.541661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-24ff6f95-83ea-41c0-aa8b-929c09577aee None None] MFA Rules not processed for user `dad93d99b865407aa120035015f4692a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:43.543089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-24ff6f95-83ea-41c0-aa8b-929c09577aee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:43.555141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:43.618966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1289/2566] 10.222.0.25 () {58 vars in 1018 bytes} [Sun Oct 2 00:59:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:43.629616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483bc62e-9ef4-422a-ad04-7d0a01ed9010 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:43.643101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483bc62e-9ef4-422a-ad04-7d0a01ed9010 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:43.649027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed788d4-0904-423c-a1cc-086296bef114 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:43.650088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed788d4-0904-423c-a1cc-086296bef114 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:43.650628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed788d4-0904-423c-a1cc-086296bef114 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:43.651127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed788d4-0904-423c-a1cc-086296bef114 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:43.663902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed788d4-0904-423c-a1cc-086296bef114 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:43.665370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed788d4-0904-423c-a1cc-086296bef114 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:43.677272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1278/2567] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:50.357010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ff6f95-83ea-41c0-aa8b-929c09577aee None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:50.358692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-24ff6f95-83ea-41c0-aa8b-929c09577aee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:50.374591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ff6f95-83ea-41c0-aa8b-929c09577aee None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:50.387637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ba60de-90be-479e-b081-3ea5d7fb75e3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:50.388928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ba60de-90be-479e-b081-3ea5d7fb75e3 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:50.389058 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ba60de-90be-479e-b081-3ea5d7fb75e3 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:50.389304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ba60de-90be-479e-b081-3ea5d7fb75e3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:50.391389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a8ba60de-90be-479e-b081-3ea5d7fb75e3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:50.456248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ba60de-90be-479e-b081-3ea5d7fb75e3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:50.457832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ba60de-90be-479e-b081-3ea5d7fb75e3 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:50.467471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1290/2568] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:50 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:52.863226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed788d4-0904-423c-a1cc-086296bef114 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:52.922711 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:52.928893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed788d4-0904-423c-a1cc-086296bef114 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:52.934004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:52.934657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:52.934936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:52.935201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] PATH_INFO: `/v3/users/2e7628060b164b26b7af194993fab104` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:52.936341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e7628060b164b26b7af194993fab104)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:52.938332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:52.984297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e7628060b164b26b7af194993fab104'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:52.984952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e7628060b164b26b7af194993fab104'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:52.992215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e7628060b164b26b7af194993fab104'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:52.992844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e7628060b164b26b7af194993fab104'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:52.999359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e7628060b164b26b7af194993fab104'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.016231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1279/2569] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:59:52 2022] DELETE /identity/v3/users/2e7628060b164b26b7af194993fab104 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:53.020707 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd5898b-5e2f-4bc6-89ad-d5efaf47d5d3 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:53.021181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd5898b-5e2f-4bc6-89ad-d5efaf47d5d3 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:53.021273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd5898b-5e2f-4bc6-89ad-d5efaf47d5d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:53.041458 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-abd5898b-5e2f-4bc6-89ad-d5efaf47d5d3 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:53.042682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-abd5898b-5e2f-4bc6-89ad-d5efaf47d5d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:53.099979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1291/2570] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:53.109838 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:53.111256 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:53.168187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bbab1b-859c-4fd1-b96d-b6919ecc048b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:53.171924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1145ab93-a9d2-4380-9069-4b34d12b1eb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:53.172523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1145ab93-a9d2-4380-9069-4b34d12b1eb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:53.172795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1145ab93-a9d2-4380-9069-4b34d12b1eb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:53.173063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1145ab93-a9d2-4380-9069-4b34d12b1eb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:53.182103 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1145ab93-a9d2-4380-9069-4b34d12b1eb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:53.183512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1145ab93-a9d2-4380-9069-4b34d12b1eb7 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:53.193039 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1280/2571] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:53.525608 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd5898b-5e2f-4bc6-89ad-d5efaf47d5d3 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:53.527072 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-abd5898b-5e2f-4bc6-89ad-d5efaf47d5d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:53.538665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd5898b-5e2f-4bc6-89ad-d5efaf47d5d3 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:53.543240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:53.543840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:53.544115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:53.544384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] PATH_INFO: `/v3/projects/12c87799cd9a4b36bead449c22bafa59` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:53.545490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] RBAC: Authorizing `identity:delete_project(project_id=12c87799cd9a4b36bead449c22bafa59)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:53.547409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:53.589631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:53.596133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '12c87799cd9a4b36bead449c22bafa59', 'request_id': 'req-76f9112d-76cc-4ee5-ac38-7b582edfb035'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.602764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '12c87799cd9a4b36bead449c22bafa59', 'request_id': 'req-76f9112d-76cc-4ee5-ac38-7b582edfb035'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.608826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '12c87799cd9a4b36bead449c22bafa59', 'request_id': 'req-76f9112d-76cc-4ee5-ac38-7b582edfb035'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.621184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] The token cache is being invalidate because project 12c87799cd9a4b36bead449c22bafa59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:59:53.621463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.622105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.623558 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1292/2572] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:59:53 2022] DELETE /identity/v3/projects/12c87799cd9a4b36bead449c22bafa59 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:53.627493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1145ab93-a9d2-4380-9069-4b34d12b1eb7 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:53.637904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1145ab93-a9d2-4380-9069-4b34d12b1eb7 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:53.694140 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:53.694875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:53.695155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:53.695488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] PATH_INFO: `/v3/users/3357f84452d34260b7eaeb9b8f28f9f7` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:53.697126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=3357f84452d34260b7eaeb9b8f28f9f7)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:53.699601 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:53.723209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:53.752073 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3357f84452d34260b7eaeb9b8f28f9f7'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.752673 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3357f84452d34260b7eaeb9b8f28f9f7'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.762557 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3357f84452d34260b7eaeb9b8f28f9f7'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.763494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3357f84452d34260b7eaeb9b8f28f9f7'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.772313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d016980e-c6a6-4b4f-92b9-46f8d75ca4ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3357f84452d34260b7eaeb9b8f28f9f7'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:53.792137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1281/2573] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:59:53 2022] DELETE /identity/v3/users/3357f84452d34260b7eaeb9b8f28f9f7 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:54.201678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:54.203216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:54.220917 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f9112d-76cc-4ee5-ac38-7b582edfb035 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:54.281652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:54.282322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:54.282631 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:54.283070 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] PATH_INFO: `/v3/projects/b76f489efaaa48cf89b90001061d7741` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:54.284487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b76f489efaaa48cf89b90001061d7741)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:54.286475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:54.332956 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b76f489efaaa48cf89b90001061d7741', 'request_id': 'req-c634e59c-207d-4396-a056-e738a7153459'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:54.341535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b76f489efaaa48cf89b90001061d7741', 'request_id': 'req-c634e59c-207d-4396-a056-e738a7153459'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:54.348027 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b76f489efaaa48cf89b90001061d7741', 'request_id': 'req-c634e59c-207d-4396-a056-e738a7153459'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:54.363982 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] The token cache is being invalidate because project b76f489efaaa48cf89b90001061d7741 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:59:54.364113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:54.364915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-c634e59c-207d-4396-a056-e738a7153459 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:54.367066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1293/2574] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:59:54 2022] DELETE /identity/v3/projects/b76f489efaaa48cf89b90001061d7741 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:54.371966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2ca563-030d-4b17-bc57-f627ca254e4d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:54.372458 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2ca563-030d-4b17-bc57-f627ca254e4d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:54.372804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb2ca563-030d-4b17-bc57-f627ca254e4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:54.394429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-fb2ca563-030d-4b17-bc57-f627ca254e4d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:54.395558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fb2ca563-030d-4b17-bc57-f627ca254e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:54.463870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1282/2575] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:54.471713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f39fd94-1131-4360-934a-89c621ce437c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:54.472016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f39fd94-1131-4360-934a-89c621ce437c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:54.472270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f39fd94-1131-4360-934a-89c621ce437c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:54.494270 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3f39fd94-1131-4360-934a-89c621ce437c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:54.495543 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3f39fd94-1131-4360-934a-89c621ce437c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:54.505614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1294/2576] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:54.510356 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2ca563-030d-4b17-bc57-f627ca254e4d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:54.522000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb2ca563-030d-4b17-bc57-f627ca254e4d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:54.527614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b76d5c-3789-4512-80d5-2dd28688bfaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:54.528231 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b76d5c-3789-4512-80d5-2dd28688bfaa admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:54.528511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b76d5c-3789-4512-80d5-2dd28688bfaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:54.528786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b76d5c-3789-4512-80d5-2dd28688bfaa admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:54.529559 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b76d5c-3789-4512-80d5-2dd28688bfaa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:59:54.529869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b76d5c-3789-4512-80d5-2dd28688bfaa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:54.531528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b76d5c-3789-4512-80d5-2dd28688bfaa admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:54.544031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1283/2577] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:59:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:59:54.548592 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f39fd94-1131-4360-934a-89c621ce437c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:54.557598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f39fd94-1131-4360-934a-89c621ce437c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:54.562848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035f63ba-1d99-4026-8c7d-c586b1c381c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:54.563459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035f63ba-1d99-4026-8c7d-c586b1c381c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:54.563735 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035f63ba-1d99-4026-8c7d-c586b1c381c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:54.564006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035f63ba-1d99-4026-8c7d-c586b1c381c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:54.564648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035f63ba-1d99-4026-8c7d-c586b1c381c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:54.566781 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035f63ba-1d99-4026-8c7d-c586b1c381c4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:54.594984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1295/2578] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:59:54 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:54.599261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b76d5c-3789-4512-80d5-2dd28688bfaa admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:54.609080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b76d5c-3789-4512-80d5-2dd28688bfaa admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:54.666521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f712b47-483f-4e39-85e2-750f3aedcccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:54.667137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f712b47-483f-4e39-85e2-750f3aedcccf admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:54.667413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f712b47-483f-4e39-85e2-750f3aedcccf admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:54.667685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f712b47-483f-4e39-85e2-750f3aedcccf admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:54.668328 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f712b47-483f-4e39-85e2-750f3aedcccf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:54.670282 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f712b47-483f-4e39-85e2-750f3aedcccf admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:54.691604 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1284/2579] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:59:54 2022] POST /identity/v3/users => generated 286 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:54.696226 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035f63ba-1d99-4026-8c7d-c586b1c381c4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:54.706291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035f63ba-1d99-4026-8c7d-c586b1c381c4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:54.710810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2e812b-b7a3-45a9-bc3b-8b6487701440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:54.711954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2e812b-b7a3-45a9-bc3b-8b6487701440 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:54.712225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2e812b-b7a3-45a9-bc3b-8b6487701440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:54.712492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2e812b-b7a3-45a9-bc3b-8b6487701440 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:54.713122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2e812b-b7a3-45a9-bc3b-8b6487701440 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:54.714734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2e812b-b7a3-45a9-bc3b-8b6487701440 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:54.727665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1296/2580] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:54.728906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f712b47-483f-4e39-85e2-750f3aedcccf admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:54.739365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f712b47-483f-4e39-85e2-750f3aedcccf admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:54.744513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dab2d7c-e2cf-4b54-bd74-4c35036c9dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:54.745110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dab2d7c-e2cf-4b54-bd74-4c35036c9dde admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:54.745389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dab2d7c-e2cf-4b54-bd74-4c35036c9dde admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:54.745737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dab2d7c-e2cf-4b54-bd74-4c35036c9dde admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:54.746472 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dab2d7c-e2cf-4b54-bd74-4c35036c9dde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:54.748084 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dab2d7c-e2cf-4b54-bd74-4c35036c9dde admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:54.760838 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1285/2581] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:54 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:54.764944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2e812b-b7a3-45a9-bc3b-8b6487701440 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:54.775332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2e812b-b7a3-45a9-bc3b-8b6487701440 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:54.780208 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da9830-917f-4097-8c6d-e7d403cf7a4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:54.780850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da9830-917f-4097-8c6d-e7d403cf7a4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:54.781120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da9830-917f-4097-8c6d-e7d403cf7a4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:54.781390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da9830-917f-4097-8c6d-e7d403cf7a4b admin admin] PATH_INFO: `/v3/projects/2205394548144ca082e6999a48b627fd/users/39428f3977604a6497d9804e9fab133d/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:54.796655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93da9830-917f-4097-8c6d-e7d403cf7a4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=2205394548144ca082e6999a48b627fd, user_id=39428f3977604a6497d9804e9fab133d, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:54.799388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93da9830-917f-4097-8c6d-e7d403cf7a4b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:54.802637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:54.814523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1297/2582] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:59:54 2022] PUT /identity/v3/projects/2205394548144ca082e6999a48b627fd/users/39428f3977604a6497d9804e9fab133d/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:54.826198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702b3f1c-6fe0-4c5b-a00b-ed33562c44ba None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:54.826487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702b3f1c-6fe0-4c5b-a00b-ed33562c44ba None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:54.826762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702b3f1c-6fe0-4c5b-a00b-ed33562c44ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:54.833841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:54.857995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-702b3f1c-6fe0-4c5b-a00b-ed33562c44ba None None] MFA Rules not processed for user `39428f3977604a6497d9804e9fab133d`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:54.859265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-702b3f1c-6fe0-4c5b-a00b-ed33562c44ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:54.933695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1286/2583] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:54 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:54.945424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da9830-917f-4097-8c6d-e7d403cf7a4b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:54.948584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-93da9830-917f-4097-8c6d-e7d403cf7a4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:55.014239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da9830-917f-4097-8c6d-e7d403cf7a4b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:55.018605 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e48966-ddbf-4f1f-adcf-ca5035cc8ad8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:55.019215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e48966-ddbf-4f1f-adcf-ca5035cc8ad8 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:55.019490 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e48966-ddbf-4f1f-adcf-ca5035cc8ad8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:55.019764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e48966-ddbf-4f1f-adcf-ca5035cc8ad8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:55.037054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e48966-ddbf-4f1f-adcf-ca5035cc8ad8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:55.038673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e48966-ddbf-4f1f-adcf-ca5035cc8ad8 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:55.058274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1298/2584] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:54 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:55.074792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702b3f1c-6fe0-4c5b-a00b-ed33562c44ba None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:55.144025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702b3f1c-6fe0-4c5b-a00b-ed33562c44ba None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:55.149558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:55.150449 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:55.150920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:55.151373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] PATH_INFO: `/v3/users/39428f3977604a6497d9804e9fab133d` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:55.152931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=39428f3977604a6497d9804e9fab133d)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:55.155150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:55.214439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39428f3977604a6497d9804e9fab133d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:55.215232 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39428f3977604a6497d9804e9fab133d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:55.223428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39428f3977604a6497d9804e9fab133d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:55.224124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39428f3977604a6497d9804e9fab133d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:55.234761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39428f3977604a6497d9804e9fab133d'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:55.256882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1287/2585] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:59:55 2022] DELETE /identity/v3/users/39428f3977604a6497d9804e9fab133d => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:55.261537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b579bb82-01de-4e2f-80cb-5532a2535232 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:55.262495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b579bb82-01de-4e2f-80cb-5532a2535232 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:55.262856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b579bb82-01de-4e2f-80cb-5532a2535232 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:55.284137 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-b579bb82-01de-4e2f-80cb-5532a2535232 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:55.285552 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b579bb82-01de-4e2f-80cb-5532a2535232 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:55.372775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1299/2586] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:55.384854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:55.386462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:55.405460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:55.457906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178cb8e2-a89e-4728-8808-1678f1293e92 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:55.462394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5096c0a-ddb9-406c-9097-60f28be9d6e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:55.463010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5096c0a-ddb9-406c-9097-60f28be9d6e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:55.463289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5096c0a-ddb9-406c-9097-60f28be9d6e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:55.463560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5096c0a-ddb9-406c-9097-60f28be9d6e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:55.473044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5096c0a-ddb9-406c-9097-60f28be9d6e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:55.474636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5096c0a-ddb9-406c-9097-60f28be9d6e8 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:55.488699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1288/2587] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:55.792509 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b579bb82-01de-4e2f-80cb-5532a2535232 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:55.793997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b579bb82-01de-4e2f-80cb-5532a2535232 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:55.806293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b579bb82-01de-4e2f-80cb-5532a2535232 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:55.816693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:55.817309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:55.817589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:55.818002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] PATH_INFO: `/v3/projects/2205394548144ca082e6999a48b627fd` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:55.819166 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2205394548144ca082e6999a48b627fd)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:55.821291 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:55.872548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2205394548144ca082e6999a48b627fd', 'request_id': 'req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:55.880807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2205394548144ca082e6999a48b627fd', 'request_id': 'req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:55.887782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2205394548144ca082e6999a48b627fd', 'request_id': 'req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:55.899908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] The token cache is being invalidate because project 2205394548144ca082e6999a48b627fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:59:55.900115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:55.900783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-e63ab38a-fc57-4a95-8f6b-c2c8ad0f9229 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:55.902561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1300/2588] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:59:55 2022] DELETE /identity/v3/projects/2205394548144ca082e6999a48b627fd => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:55.907658 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcfb4a2-e5f1-4530-9d01-94a669195713 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:55.907887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcfb4a2-e5f1-4530-9d01-94a669195713 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:55.908452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcfb4a2-e5f1-4530-9d01-94a669195713 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:55.930655 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-cbcfb4a2-e5f1-4530-9d01-94a669195713 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:55.931861 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cbcfb4a2-e5f1-4530-9d01-94a669195713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:55.993519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1289/2589] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:56.002660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2aaad1-5e2f-4475-bc2b-4fa1a64536c6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.002946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2aaad1-5e2f-4475-bc2b-4fa1a64536c6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.003275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2aaad1-5e2f-4475-bc2b-4fa1a64536c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.023885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4a2aaad1-5e2f-4475-bc2b-4fa1a64536c6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:56.025118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4a2aaad1-5e2f-4475-bc2b-4fa1a64536c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:56.040053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1301/2590] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:56.044563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcfb4a2-e5f1-4530-9d01-94a669195713 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:56.054584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcfb4a2-e5f1-4530-9d01-94a669195713 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:56.059623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476f228c-d7b7-45cf-9533-12dfdcb1a881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:56.060225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476f228c-d7b7-45cf-9533-12dfdcb1a881 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.060538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476f228c-d7b7-45cf-9533-12dfdcb1a881 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.060800 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476f228c-d7b7-45cf-9533-12dfdcb1a881 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.061596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476f228c-d7b7-45cf-9533-12dfdcb1a881 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:59:56.061907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476f228c-d7b7-45cf-9533-12dfdcb1a881 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:56.063663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476f228c-d7b7-45cf-9533-12dfdcb1a881 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:56.077201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1290/2591] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:59:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 00:59:56.081941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2aaad1-5e2f-4475-bc2b-4fa1a64536c6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:56.094386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2aaad1-5e2f-4475-bc2b-4fa1a64536c6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:56.099273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcd6bae-caae-4ac0-b295-a52d8b098e74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:56.099914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd6bae-caae-4ac0-b295-a52d8b098e74 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.100233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd6bae-caae-4ac0-b295-a52d8b098e74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.100536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcd6bae-caae-4ac0-b295-a52d8b098e74 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.101239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcd6bae-caae-4ac0-b295-a52d8b098e74 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:56.103326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcd6bae-caae-4ac0-b295-a52d8b098e74 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:56.119376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:56.132548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1302/2592] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:59:56 2022] POST /identity/v3/projects => generated 359 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:56.137410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476f228c-d7b7-45cf-9533-12dfdcb1a881 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:56.147144 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476f228c-d7b7-45cf-9533-12dfdcb1a881 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:56.202471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5575a4ac-b336-4300-aa21-ca5d1ba2556a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:56.203000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5575a4ac-b336-4300-aa21-ca5d1ba2556a admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.203280 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5575a4ac-b336-4300-aa21-ca5d1ba2556a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.203556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5575a4ac-b336-4300-aa21-ca5d1ba2556a admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.204189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5575a4ac-b336-4300-aa21-ca5d1ba2556a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:56.206121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5575a4ac-b336-4300-aa21-ca5d1ba2556a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:56.227813 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1291/2593] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:59:56 2022] POST /identity/v3/users => generated 280 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:56.231273 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcd6bae-caae-4ac0-b295-a52d8b098e74 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:56.241354 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcd6bae-caae-4ac0-b295-a52d8b098e74 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:56.246457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1205d5aa-d2a4-44fa-aabf-c67dab025278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:56.247109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1205d5aa-d2a4-44fa-aabf-c67dab025278 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.247411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1205d5aa-d2a4-44fa-aabf-c67dab025278 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.247703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1205d5aa-d2a4-44fa-aabf-c67dab025278 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.248383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1205d5aa-d2a4-44fa-aabf-c67dab025278 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:56.250171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1205d5aa-d2a4-44fa-aabf-c67dab025278 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:56.261495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1303/2594] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:56.265699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5575a4ac-b336-4300-aa21-ca5d1ba2556a admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:56.276736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5575a4ac-b336-4300-aa21-ca5d1ba2556a admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:56.281289 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7aeb385-b411-4a71-8105-c8df56c12ac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:56.281882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7aeb385-b411-4a71-8105-c8df56c12ac0 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.282205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7aeb385-b411-4a71-8105-c8df56c12ac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.282485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7aeb385-b411-4a71-8105-c8df56c12ac0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.283116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7aeb385-b411-4a71-8105-c8df56c12ac0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:56.284705 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7aeb385-b411-4a71-8105-c8df56c12ac0 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:56.295099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1292/2595] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:56.299078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1205d5aa-d2a4-44fa-aabf-c67dab025278 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:56.308995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1205d5aa-d2a4-44fa-aabf-c67dab025278 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:56.313594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98de05b-4fbd-4989-870a-da202e43595a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:56.314262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98de05b-4fbd-4989-870a-da202e43595a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.314539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98de05b-4fbd-4989-870a-da202e43595a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.314810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98de05b-4fbd-4989-870a-da202e43595a admin admin] PATH_INFO: `/v3/projects/3879c0f3ba2646269fcd5855dcd2201a/users/7f405014413345cabe6bdd2ace73f15f/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.329374 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98de05b-4fbd-4989-870a-da202e43595a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3879c0f3ba2646269fcd5855dcd2201a, user_id=7f405014413345cabe6bdd2ace73f15f, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:56.332319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98de05b-4fbd-4989-870a-da202e43595a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:56.347040 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1304/2596] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:59:56 2022] PUT /identity/v3/projects/3879c0f3ba2646269fcd5855dcd2201a/users/7f405014413345cabe6bdd2ace73f15f/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 00:59:56.357240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74300bbf-9350-41f4-be2d-03ba2e0d8896 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.357526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74300bbf-9350-41f4-be2d-03ba2e0d8896 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.357809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74300bbf-9350-41f4-be2d-03ba2e0d8896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.385731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-74300bbf-9350-41f4-be2d-03ba2e0d8896 None None] MFA Rules not processed for user `7f405014413345cabe6bdd2ace73f15f`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:56.386969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-74300bbf-9350-41f4-be2d-03ba2e0d8896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:56.395297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:56.458504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1293/2597] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:56 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:56.493467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98de05b-4fbd-4989-870a-da202e43595a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:56.495194 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-d98de05b-4fbd-4989-870a-da202e43595a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:56.549437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98de05b-4fbd-4989-870a-da202e43595a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:56.553109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b781601-8658-4e8e-a73f-46e28c2c465b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:56.553726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b781601-8658-4e8e-a73f-46e28c2c465b service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.554000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b781601-8658-4e8e-a73f-46e28c2c465b service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.554298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b781601-8658-4e8e-a73f-46e28c2c465b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.563035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b781601-8658-4e8e-a73f-46e28c2c465b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:56.564448 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b781601-8658-4e8e-a73f-46e28c2c465b service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:56.573694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1305/2598] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:56 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:56.588840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74300bbf-9350-41f4-be2d-03ba2e0d8896 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:56.649200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74300bbf-9350-41f4-be2d-03ba2e0d8896 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:56.653691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:56.654332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.654612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.654885 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] PATH_INFO: `/v3/users/7f405014413345cabe6bdd2ace73f15f` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.655995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f405014413345cabe6bdd2ace73f15f)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:56.657907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:56.700928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f405014413345cabe6bdd2ace73f15f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:56.701579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f405014413345cabe6bdd2ace73f15f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:56.708237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f405014413345cabe6bdd2ace73f15f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:56.708928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f405014413345cabe6bdd2ace73f15f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:56.715222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f405014413345cabe6bdd2ace73f15f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:56.732673 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1294/2599] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 00:59:56 2022] DELETE /identity/v3/users/7f405014413345cabe6bdd2ace73f15f => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:56.737149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd839096-6414-44db-9d4e-d79af6c559cb None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.737426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd839096-6414-44db-9d4e-d79af6c559cb None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.737717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd839096-6414-44db-9d4e-d79af6c559cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.757118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-dd839096-6414-44db-9d4e-d79af6c559cb None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:56.758994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-dd839096-6414-44db-9d4e-d79af6c559cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:56.817843 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1306/2600] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:56.827571 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:56.828963 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:56.862537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:56.890035 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d293bded-0c21-4600-98b4-97e30de8928b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:56.893697 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedfd600-8bbe-42ad-be1b-cab04395af91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:56.894326 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedfd600-8bbe-42ad-be1b-cab04395af91 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:56.894622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedfd600-8bbe-42ad-be1b-cab04395af91 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:56.895009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedfd600-8bbe-42ad-be1b-cab04395af91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:56.904141 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedfd600-8bbe-42ad-be1b-cab04395af91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:56.905558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedfd600-8bbe-42ad-be1b-cab04395af91 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:56.915752 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1295/2601] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 00:59:57.177758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd839096-6414-44db-9d4e-d79af6c559cb None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:57.179349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-dd839096-6414-44db-9d4e-d79af6c559cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:57.191779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd839096-6414-44db-9d4e-d79af6c559cb None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:57.196613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:57.197238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.197512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.197779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] PATH_INFO: `/v3/projects/3879c0f3ba2646269fcd5855dcd2201a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.198945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] RBAC: Authorizing `identity:delete_project(project_id=3879c0f3ba2646269fcd5855dcd2201a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:57.200868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:57.247442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3879c0f3ba2646269fcd5855dcd2201a', 'request_id': 'req-8da7aa77-2143-41c2-b595-340f765b3bcf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:57.254434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3879c0f3ba2646269fcd5855dcd2201a', 'request_id': 'req-8da7aa77-2143-41c2-b595-340f765b3bcf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:57.255547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=3,keystone:SELECT=92,keystone:DELETE=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:57.263033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3879c0f3ba2646269fcd5855dcd2201a', 'request_id': 'req-8da7aa77-2143-41c2-b595-340f765b3bcf'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:57.273854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] The token cache is being invalidate because project 3879c0f3ba2646269fcd5855dcd2201a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 00:59:57.274176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:57.275122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8da7aa77-2143-41c2-b595-340f765b3bcf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 00:59:57.278972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1307/2602] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 00:59:57 2022] DELETE /identity/v3/projects/3879c0f3ba2646269fcd5855dcd2201a => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:57.283892 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430d7fa1-3fc1-453c-a4a2-ebbc7d99e621 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.284175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430d7fa1-3fc1-453c-a4a2-ebbc7d99e621 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.284446 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430d7fa1-3fc1-453c-a4a2-ebbc7d99e621 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.304636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-430d7fa1-3fc1-453c-a4a2-ebbc7d99e621 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:57.305815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-430d7fa1-3fc1-453c-a4a2-ebbc7d99e621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:57.362867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1296/2603] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:57.370931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f9963-2252-4b95-ac0d-0b755f834598 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.371385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f9963-2252-4b95-ac0d-0b755f834598 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.371574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f9963-2252-4b95-ac0d-0b755f834598 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.390939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-601f9963-2252-4b95-ac0d-0b755f834598 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:57.392133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-601f9963-2252-4b95-ac0d-0b755f834598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:57.400650 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1308/2604] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:57.404948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430d7fa1-3fc1-453c-a4a2-ebbc7d99e621 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:57.414956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430d7fa1-3fc1-453c-a4a2-ebbc7d99e621 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:57.419386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5e2703-7e18-40d2-a301-6910908e4039 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:57.419972 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5e2703-7e18-40d2-a301-6910908e4039 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.420247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5e2703-7e18-40d2-a301-6910908e4039 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.420515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5e2703-7e18-40d2-a301-6910908e4039 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.421147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5e2703-7e18-40d2-a301-6910908e4039 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 00:59:57.421442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5e2703-7e18-40d2-a301-6910908e4039 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:57.423043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5e2703-7e18-40d2-a301-6910908e4039 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:57.434128 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1297/2605] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 00:59:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 00:59:57.438380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601f9963-2252-4b95-ac0d-0b755f834598 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:57.447507 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601f9963-2252-4b95-ac0d-0b755f834598 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:57.452051 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cc20cc-42b4-44b7-a9ff-867952d41332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:57.452634 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cc20cc-42b4-44b7-a9ff-867952d41332 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.452906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cc20cc-42b4-44b7-a9ff-867952d41332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.453171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cc20cc-42b4-44b7-a9ff-867952d41332 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.453794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cc20cc-42b4-44b7-a9ff-867952d41332 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:57.455724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cc20cc-42b4-44b7-a9ff-867952d41332 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:57.483357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1309/2606] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 00:59:57 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:57.487448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5e2703-7e18-40d2-a301-6910908e4039 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:57.497357 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5e2703-7e18-40d2-a301-6910908e4039 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:57.553450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b047b430-ad48-4b94-85e9-c0cc523388e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:57.554052 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b047b430-ad48-4b94-85e9-c0cc523388e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.554359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b047b430-ad48-4b94-85e9-c0cc523388e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.554630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b047b430-ad48-4b94-85e9-c0cc523388e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.555251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b047b430-ad48-4b94-85e9-c0cc523388e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:57.557127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b047b430-ad48-4b94-85e9-c0cc523388e9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:57.578081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1298/2607] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 00:59:57 2022] POST /identity/v3/users => generated 287 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 00:59:57.582143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cc20cc-42b4-44b7-a9ff-867952d41332 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:57.593330 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cc20cc-42b4-44b7-a9ff-867952d41332 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:57.597947 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd01bfa-aec7-43cd-a840-02660db20d4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:57.598575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd01bfa-aec7-43cd-a840-02660db20d4f admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.598851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd01bfa-aec7-43cd-a840-02660db20d4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.599122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd01bfa-aec7-43cd-a840-02660db20d4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.599854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd01bfa-aec7-43cd-a840-02660db20d4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:57.601476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd01bfa-aec7-43cd-a840-02660db20d4f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:57.611560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1310/2608] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:57.615384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b047b430-ad48-4b94-85e9-c0cc523388e9 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:57.625659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b047b430-ad48-4b94-85e9-c0cc523388e9 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:57.630248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93b1f32-020a-4686-9616-52ec02f5b772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:57.630851 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93b1f32-020a-4686-9616-52ec02f5b772 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.631129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93b1f32-020a-4686-9616-52ec02f5b772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.631399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93b1f32-020a-4686-9616-52ec02f5b772 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.632033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93b1f32-020a-4686-9616-52ec02f5b772 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:57.634167 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93b1f32-020a-4686-9616-52ec02f5b772 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:57.646200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1299/2609] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 00:59:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 00:59:57.650474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd01bfa-aec7-43cd-a840-02660db20d4f admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:57.660441 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd01bfa-aec7-43cd-a840-02660db20d4f admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:57.666046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c866d8a5-d8ae-4716-b0d9-6e9ff7fcc35d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:57.666857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c866d8a5-d8ae-4716-b0d9-6e9ff7fcc35d admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.667131 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c866d8a5-d8ae-4716-b0d9-6e9ff7fcc35d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.667401 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c866d8a5-d8ae-4716-b0d9-6e9ff7fcc35d admin admin] PATH_INFO: `/v3/projects/3b4519c281444cec82120fc744a4c5f0/users/0df9426abb5e495ea34429a5cccf9fe4/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.682470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c866d8a5-d8ae-4716-b0d9-6e9ff7fcc35d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b4519c281444cec82120fc744a4c5f0, user_id=0df9426abb5e495ea34429a5cccf9fe4, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:57.685072 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c866d8a5-d8ae-4716-b0d9-6e9ff7fcc35d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:57.698410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1311/2610] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 00:59:57 2022] PUT /identity/v3/projects/3b4519c281444cec82120fc744a4c5f0/users/0df9426abb5e495ea34429a5cccf9fe4/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 00:59:57.703406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce6770-d079-4407-83b1-3fca30839fbb None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.703693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce6770-d079-4407-83b1-3fca30839fbb None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.703968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fce6770-d079-4407-83b1-3fca30839fbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.723014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-0fce6770-d079-4407-83b1-3fca30839fbb None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:57.724201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-0fce6770-d079-4407-83b1-3fca30839fbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:57.759049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 00:59:57.786583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1300/2611] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:57.796320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c866d8a5-d8ae-4716-b0d9-6e9ff7fcc35d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:57.798127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c866d8a5-d8ae-4716-b0d9-6e9ff7fcc35d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:57.853598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c866d8a5-d8ae-4716-b0d9-6e9ff7fcc35d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:57.857174 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650ee856-0a9d-4275-86b7-7bb7eee7277e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:57.857762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650ee856-0a9d-4275-86b7-7bb7eee7277e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:57.858037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650ee856-0a9d-4275-86b7-7bb7eee7277e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:57.858340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650ee856-0a9d-4275-86b7-7bb7eee7277e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:57.867446 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650ee856-0a9d-4275-86b7-7bb7eee7277e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:57.868855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650ee856-0a9d-4275-86b7-7bb7eee7277e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:57.878027 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1312/2612] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 00:59:58.562732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411d2b04-b6d5-44ec-92ac-abb675363315 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:58.563243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411d2b04-b6d5-44ec-92ac-abb675363315 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:58.563356 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411d2b04-b6d5-44ec-92ac-abb675363315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:58.591393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-411d2b04-b6d5-44ec-92ac-abb675363315 None None] MFA Rules not processed for user `0df9426abb5e495ea34429a5cccf9fe4`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 00:59:58.592529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-411d2b04-b6d5-44ec-92ac-abb675363315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:58.697438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1301/2613] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 00:59:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 00:59:58.707869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650ee856-0a9d-4275-86b7-7bb7eee7277e service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 00:59:58.709461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-650ee856-0a9d-4275-86b7-7bb7eee7277e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 00:59:58.771546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650ee856-0a9d-4275-86b7-7bb7eee7277e service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 00:59:58.775912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810e3118-b8f1-469f-adae-841fdba83215 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 00:59:58.776504 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810e3118-b8f1-469f-adae-841fdba83215 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 00:59:58.776775 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810e3118-b8f1-469f-adae-841fdba83215 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 00:59:58.777041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810e3118-b8f1-469f-adae-841fdba83215 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 00:59:58.785274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810e3118-b8f1-469f-adae-841fdba83215 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 00:59:58.786732 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810e3118-b8f1-469f-adae-841fdba83215 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 00:59:58.796493 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1313/2614] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 00:59:58 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:00:00.195077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b221ed74-56d3-47fb-af13-3d320ed29a66 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:00.195550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b221ed74-56d3-47fb-af13-3d320ed29a66 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:00.195645 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b221ed74-56d3-47fb-af13-3d320ed29a66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:00.214733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-b221ed74-56d3-47fb-af13-3d320ed29a66 None None] MFA Rules not processed for user `0df9426abb5e495ea34429a5cccf9fe4`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:00:00.215924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b221ed74-56d3-47fb-af13-3d320ed29a66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:00.222755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1302/2615] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:00:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:00:00.472915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810e3118-b8f1-469f-adae-841fdba83215 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:00:00.482175 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810e3118-b8f1-469f-adae-841fdba83215 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:00:00.485811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5c7db8-98c4-4485-9959-52436871efb9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:00:00.486439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5c7db8-98c4-4485-9959-52436871efb9 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:00.486871 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5c7db8-98c4-4485-9959-52436871efb9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:00.487144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5c7db8-98c4-4485-9959-52436871efb9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:00.488853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-dd5c7db8-98c4-4485-9959-52436871efb9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:00.499646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5c7db8-98c4-4485-9959-52436871efb9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:00:00.501067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5c7db8-98c4-4485-9959-52436871efb9 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:00:00.509533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1314/2616] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:00:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:00:01.116197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b221ed74-56d3-47fb-af13-3d320ed29a66 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:00:01.119243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-b221ed74-56d3-47fb-af13-3d320ed29a66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:01.256732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b221ed74-56d3-47fb-af13-3d320ed29a66 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:00:01.264227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c576e6a6-2088-464f-863c-11ac1b671f2f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:00:01.264999 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c576e6a6-2088-464f-863c-11ac1b671f2f service placement] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:01.265389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c576e6a6-2088-464f-863c-11ac1b671f2f service placement] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:01.265736 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c576e6a6-2088-464f-863c-11ac1b671f2f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:01.268355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c576e6a6-2088-464f-863c-11ac1b671f2f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:01.280195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c576e6a6-2088-464f-863c-11ac1b671f2f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:00:01.281860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c576e6a6-2088-464f-863c-11ac1b671f2f service placement] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:00:01.293431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1303/2617] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:00:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 178 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:00:01.344662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5c7db8-98c4-4485-9959-52436871efb9 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:00:01.353640 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5c7db8-98c4-4485-9959-52436871efb9 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:00:01.357519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3909b5a-7073-4cf3-93f4-01dd5ecd0669 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:00:01.358218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3909b5a-7073-4cf3-93f4-01dd5ecd0669 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:01.358582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3909b5a-7073-4cf3-93f4-01dd5ecd0669 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:01.358853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3909b5a-7073-4cf3-93f4-01dd5ecd0669 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:01.360625 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f3909b5a-7073-4cf3-93f4-01dd5ecd0669 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:01.371633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3909b5a-7073-4cf3-93f4-01dd5ecd0669 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:00:01.373093 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3909b5a-7073-4cf3-93f4-01dd5ecd0669 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:00:01.377959 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:00:01.383364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1315/2618] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:00:01 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:00:11.288869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=77 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:00:11.448237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590a686b-c8a7-4524-9088-99ac799e3380 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:11.449111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590a686b-c8a7-4524-9088-99ac799e3380 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:11.449387 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590a686b-c8a7-4524-9088-99ac799e3380 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:11.468543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-590a686b-c8a7-4524-9088-99ac799e3380 None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:00:11.470086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-590a686b-c8a7-4524-9088-99ac799e3380 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:11.532388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1304/2619] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 01:00:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:00:11.590900 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3909b5a-7073-4cf3-93f4-01dd5ecd0669 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:00:11.592535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f3909b5a-7073-4cf3-93f4-01dd5ecd0669 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:11.669736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3909b5a-7073-4cf3-93f4-01dd5ecd0669 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:00:11.674660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487c7507-30ec-4ee9-bf7b-dbfd90e5acd0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:00:11.675260 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c7507-30ec-4ee9-bf7b-dbfd90e5acd0 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:11.675535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c7507-30ec-4ee9-bf7b-dbfd90e5acd0 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:11.675806 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c7507-30ec-4ee9-bf7b-dbfd90e5acd0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:11.685009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487c7507-30ec-4ee9-bf7b-dbfd90e5acd0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:00:11.686613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487c7507-30ec-4ee9-bf7b-dbfd90e5acd0 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:00:11.694489 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1316/2620] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 01:00:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 01:00:17.208534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590a686b-c8a7-4524-9088-99ac799e3380 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:00:17.210419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-590a686b-c8a7-4524-9088-99ac799e3380 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:17.485310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590a686b-c8a7-4524-9088-99ac799e3380 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:00:17.491958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec529f56-f74b-4b32-94e3-217b4450ddb5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:00:17.492659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec529f56-f74b-4b32-94e3-217b4450ddb5 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:17.493019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec529f56-f74b-4b32-94e3-217b4450ddb5 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:17.493330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec529f56-f74b-4b32-94e3-217b4450ddb5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:17.495757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ec529f56-f74b-4b32-94e3-217b4450ddb5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:17.513865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec529f56-f74b-4b32-94e3-217b4450ddb5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:00:17.515900 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec529f56-f74b-4b32-94e3-217b4450ddb5 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:00:17.529396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1305/2621] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:00:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:00:18.515390 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690b29d0-8fe8-4f65-9af5-bad970c779e8 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:18.515932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690b29d0-8fe8-4f65-9af5-bad970c779e8 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:18.515932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690b29d0-8fe8-4f65-9af5-bad970c779e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:18.538424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-690b29d0-8fe8-4f65-9af5-bad970c779e8 None None] MFA Rules not processed for user `0df9426abb5e495ea34429a5cccf9fe4`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:00:18.539646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-690b29d0-8fe8-4f65-9af5-bad970c779e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:18.547375 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1317/2622] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:00:18 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:00:25.052856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec529f56-f74b-4b32-94e3-217b4450ddb5 service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:00:25.054782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ec529f56-f74b-4b32-94e3-217b4450ddb5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:25.069841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec529f56-f74b-4b32-94e3-217b4450ddb5 service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:00:25.075370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c917bd-7ad4-4c2f-ae14-d9718c5a877a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:00:25.076085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c917bd-7ad4-4c2f-ae14-d9718c5a877a service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:25.076365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c917bd-7ad4-4c2f-ae14-d9718c5a877a service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:25.076639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c917bd-7ad4-4c2f-ae14-d9718c5a877a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:25.078492 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-26c917bd-7ad4-4c2f-ae14-d9718c5a877a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:25.091964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c917bd-7ad4-4c2f-ae14-d9718c5a877a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:00:25.093765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c917bd-7ad4-4c2f-ae14-d9718c5a877a service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:00:25.109738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1306/2623] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:00:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:00:28.531586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=35 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:00:28.783801 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d1d54-c7a8-4291-8ed7-81ffff706f67 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:28.784080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d1d54-c7a8-4291-8ed7-81ffff706f67 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:28.784352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6d1d54-c7a8-4291-8ed7-81ffff706f67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:28.802906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fc6d1d54-c7a8-4291-8ed7-81ffff706f67 None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:00:28.804080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fc6d1d54-c7a8-4291-8ed7-81ffff706f67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:28.811024 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1318/2624] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:00:28.823275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c917bd-7ad4-4c2f-ae14-d9718c5a877a service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:00:28.834179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c917bd-7ad4-4c2f-ae14-d9718c5a877a service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:00:28.839662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b10734-1ee9-4bcd-b848-d589c95bf7c8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:00:28.840254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b10734-1ee9-4bcd-b848-d589c95bf7c8 service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:28.840530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b10734-1ee9-4bcd-b848-d589c95bf7c8 service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:28.840799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b10734-1ee9-4bcd-b848-d589c95bf7c8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:28.848633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b10734-1ee9-4bcd-b848-d589c95bf7c8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:00:28.850209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b10734-1ee9-4bcd-b848-d589c95bf7c8 service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:00:28.858208 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1307/2625] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 01:00:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 01:00:38.798272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:00:38.854982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=71 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:00:45.933850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171435d9-6a2b-4bab-bf82-ddcabcd33d53 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:45.934349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171435d9-6a2b-4bab-bf82-ddcabcd33d53 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:45.934716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171435d9-6a2b-4bab-bf82-ddcabcd33d53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:45.964535 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-171435d9-6a2b-4bab-bf82-ddcabcd33d53 None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:00:45.965745 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-171435d9-6a2b-4bab-bf82-ddcabcd33d53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:45.972905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1319/2626] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 2 01:00:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:00:45.986747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b10734-1ee9-4bcd-b848-d589c95bf7c8 service swift] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:00:45.998627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b10734-1ee9-4bcd-b848-d589c95bf7c8 service swift] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:00:46.003712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f73780a-a4a4-447c-bf7c-b8c46279c64d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:00:46.004303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f73780a-a4a4-447c-bf7c-b8c46279c64d service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:46.004580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f73780a-a4a4-447c-bf7c-b8c46279c64d service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:46.004852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f73780a-a4a4-447c-bf7c-b8c46279c64d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:46.013023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f73780a-a4a4-447c-bf7c-b8c46279c64d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:00:46.014606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f73780a-a4a4-447c-bf7c-b8c46279c64d service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:00:46.023393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1308/2627] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 01:00:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 01:00:53.184840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617a90d5-308f-4377-a238-2e6eeb1c3bed None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:53.185302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617a90d5-308f-4377-a238-2e6eeb1c3bed None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:53.185409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617a90d5-308f-4377-a238-2e6eeb1c3bed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:53.204747 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-617a90d5-308f-4377-a238-2e6eeb1c3bed None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:00:53.205919 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-617a90d5-308f-4377-a238-2e6eeb1c3bed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:00:53.212918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1320/2628] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 01:00:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:00:53.224947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f73780a-a4a4-447c-bf7c-b8c46279c64d service swift] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:00:53.235288 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f73780a-a4a4-447c-bf7c-b8c46279c64d service swift] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:00:53.241013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5265f39c-ebca-4f98-95d3-376240fcc4be service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:00:53.241720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5265f39c-ebca-4f98-95d3-376240fcc4be service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:00:53.242053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5265f39c-ebca-4f98-95d3-376240fcc4be service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:00:53.242417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5265f39c-ebca-4f98-95d3-376240fcc4be service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:00:53.250733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5265f39c-ebca-4f98-95d3-376240fcc4be service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:00:53.252303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5265f39c-ebca-4f98-95d3-376240fcc4be service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:00:53.260534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1309/2629] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 01:00:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 01:01:03.200019 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:03.259639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=12 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:08.722444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5018868-3998-4445-9937-f21edc96255f None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:08.722965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5018868-3998-4445-9937-f21edc96255f None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:08.722965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5018868-3998-4445-9937-f21edc96255f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:08.743025 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a5018868-3998-4445-9937-f21edc96255f None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:08.744258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a5018868-3998-4445-9937-f21edc96255f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:08.751326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1321/2630] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 2 01:01:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:08.763780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5265f39c-ebca-4f98-95d3-376240fcc4be service swift] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:08.776549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5265f39c-ebca-4f98-95d3-376240fcc4be service swift] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:08.781837 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc5f8a2-10a1-490d-81f7-8bcb5e8654fd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:08.782659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc5f8a2-10a1-490d-81f7-8bcb5e8654fd service swift] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:08.783002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc5f8a2-10a1-490d-81f7-8bcb5e8654fd service swift] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:08.783393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc5f8a2-10a1-490d-81f7-8bcb5e8654fd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:08.792690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc5f8a2-10a1-490d-81f7-8bcb5e8654fd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:08.794432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc5f8a2-10a1-490d-81f7-8bcb5e8654fd service swift] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:08.803167 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1310/2631] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 01:01:08 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 02 01:01:10.702770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5018868-3998-4445-9937-f21edc96255f None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:10.715614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5018868-3998-4445-9937-f21edc96255f None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:10.720495 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a0d79f-4b2a-4022-961f-438fa262c3b3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:10.721134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a0d79f-4b2a-4022-961f-438fa262c3b3 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:10.721550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a0d79f-4b2a-4022-961f-438fa262c3b3 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:10.721846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a0d79f-4b2a-4022-961f-438fa262c3b3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:10.723654 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-49a0d79f-4b2a-4022-961f-438fa262c3b3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:10.735777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a0d79f-4b2a-4022-961f-438fa262c3b3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:10.737589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a0d79f-4b2a-4022-961f-438fa262c3b3 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:10.749667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1322/2632] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:10 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:10.765043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc5f8a2-10a1-490d-81f7-8bcb5e8654fd service swift] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:10.777214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc5f8a2-10a1-490d-81f7-8bcb5e8654fd service swift] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:10.782270 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:10.782883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:10.783156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:10.783432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] PATH_INFO: `/v3/users/0df9426abb5e495ea34429a5cccf9fe4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:10.784515 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0df9426abb5e495ea34429a5cccf9fe4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:10.786457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:10.832854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0df9426abb5e495ea34429a5cccf9fe4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:10.833498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0df9426abb5e495ea34429a5cccf9fe4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:10.840839 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0df9426abb5e495ea34429a5cccf9fe4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:10.841481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0df9426abb5e495ea34429a5cccf9fe4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:10.848799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-4858fc6a-1b87-4445-aac7-10ff983742b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0df9426abb5e495ea34429a5cccf9fe4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:10.870606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1311/2633] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:10 2022] DELETE /identity/v3/users/0df9426abb5e495ea34429a5cccf9fe4 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:11.043695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a0d79f-4b2a-4022-961f-438fa262c3b3 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:11.054331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a0d79f-4b2a-4022-961f-438fa262c3b3 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:11.110637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:11.111258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.111533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.111802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] PATH_INFO: `/v3/projects/3b4519c281444cec82120fc744a4c5f0` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.112888 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b4519c281444cec82120fc744a4c5f0)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:11.114892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:11.160295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b4519c281444cec82120fc744a4c5f0', 'request_id': 'req-2cc9af08-fca6-4c09-9079-07cc2095d012'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:11.167230 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b4519c281444cec82120fc744a4c5f0', 'request_id': 'req-2cc9af08-fca6-4c09-9079-07cc2095d012'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:11.173447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b4519c281444cec82120fc744a4c5f0', 'request_id': 'req-2cc9af08-fca6-4c09-9079-07cc2095d012'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:11.183989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] The token cache is being invalidate because project 3b4519c281444cec82120fc744a4c5f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:11.184271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:11.184884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2cc9af08-fca6-4c09-9079-07cc2095d012 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:11.186342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1323/2634] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:11 2022] DELETE /identity/v3/projects/3b4519c281444cec82120fc744a4c5f0 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:11.191272 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16f7fbf-1f95-4680-be56-398fad1f68e4 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.191559 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16f7fbf-1f95-4680-be56-398fad1f68e4 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.191832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16f7fbf-1f95-4680-be56-398fad1f68e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.211294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-c16f7fbf-1f95-4680-be56-398fad1f68e4 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:11.212475 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c16f7fbf-1f95-4680-be56-398fad1f68e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:11.270738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1312/2635] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:11.278525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b030fe6-6cd7-4835-a831-741b726fa268 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.278844 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b030fe6-6cd7-4835-a831-741b726fa268 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.279143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b030fe6-6cd7-4835-a831-741b726fa268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.298078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-7b030fe6-6cd7-4835-a831-741b726fa268 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:11.299277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b030fe6-6cd7-4835-a831-741b726fa268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:11.307442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1324/2636] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:01:11.311686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16f7fbf-1f95-4680-be56-398fad1f68e4 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:11.321643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16f7fbf-1f95-4680-be56-398fad1f68e4 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:11.326137 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b291ad-ed48-4526-8320-af82a02c486d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:11.326727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b291ad-ed48-4526-8320-af82a02c486d admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.327003 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b291ad-ed48-4526-8320-af82a02c486d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.327275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b291ad-ed48-4526-8320-af82a02c486d admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.327936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b291ad-ed48-4526-8320-af82a02c486d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:01:11.328235 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b291ad-ed48-4526-8320-af82a02c486d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:11.329799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b291ad-ed48-4526-8320-af82a02c486d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:11.340897 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1313/2637] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:01:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:01:11.345114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b030fe6-6cd7-4835-a831-741b726fa268 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:11.354773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b030fe6-6cd7-4835-a831-741b726fa268 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:11.359403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee804cf-41d3-4338-bd6b-e7019aaa511a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:11.359987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee804cf-41d3-4338-bd6b-e7019aaa511a admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.360265 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee804cf-41d3-4338-bd6b-e7019aaa511a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.360536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee804cf-41d3-4338-bd6b-e7019aaa511a admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee804cf-41d3-4338-bd6b-e7019aaa511a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:11.363085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee804cf-41d3-4338-bd6b-e7019aaa511a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:11.394239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1325/2638] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:01:11 2022] POST /identity/v3/projects => generated 363 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:11.395420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b291ad-ed48-4526-8320-af82a02c486d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:11.406166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b291ad-ed48-4526-8320-af82a02c486d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:11.451374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:11.466920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539fc943-18f0-4664-b44e-0b3274bb9932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:11.467669 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539fc943-18f0-4664-b44e-0b3274bb9932 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.467948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539fc943-18f0-4664-b44e-0b3274bb9932 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.468257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539fc943-18f0-4664-b44e-0b3274bb9932 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.468870 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539fc943-18f0-4664-b44e-0b3274bb9932 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:11.470865 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539fc943-18f0-4664-b44e-0b3274bb9932 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:11.497982 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1314/2639] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:01:11 2022] POST /identity/v3/users => generated 282 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:11.500955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee804cf-41d3-4338-bd6b-e7019aaa511a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:11.517597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee804cf-41d3-4338-bd6b-e7019aaa511a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:11.522253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc457a1d-0d9b-4804-b4b8-64ec981d068c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:11.522855 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc457a1d-0d9b-4804-b4b8-64ec981d068c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.523129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc457a1d-0d9b-4804-b4b8-64ec981d068c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.523429 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc457a1d-0d9b-4804-b4b8-64ec981d068c admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.524056 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc457a1d-0d9b-4804-b4b8-64ec981d068c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:11.525664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc457a1d-0d9b-4804-b4b8-64ec981d068c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:11.535461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1326/2640] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:11 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:11.540112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539fc943-18f0-4664-b44e-0b3274bb9932 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:11.549803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539fc943-18f0-4664-b44e-0b3274bb9932 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:11.554285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26116c83-b488-43e2-9e5b-0b90a8168d26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:11.555472 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26116c83-b488-43e2-9e5b-0b90a8168d26 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.555789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26116c83-b488-43e2-9e5b-0b90a8168d26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.556058 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26116c83-b488-43e2-9e5b-0b90a8168d26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.556683 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26116c83-b488-43e2-9e5b-0b90a8168d26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:11.558278 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26116c83-b488-43e2-9e5b-0b90a8168d26 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:11.567752 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1315/2641] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:11.571880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc457a1d-0d9b-4804-b4b8-64ec981d068c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:11.581539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc457a1d-0d9b-4804-b4b8-64ec981d068c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:11.586162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e190a4ef-b0d9-4297-a608-c61f5c088878 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:11.586795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e190a4ef-b0d9-4297-a608-c61f5c088878 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.587067 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e190a4ef-b0d9-4297-a608-c61f5c088878 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.587334 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e190a4ef-b0d9-4297-a608-c61f5c088878 admin admin] PATH_INFO: `/v3/projects/701d8b3f08cf4ba6af9f072877249d4c/users/54743d61389549f982b874edd65c497e/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.602367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e190a4ef-b0d9-4297-a608-c61f5c088878 admin admin] RBAC: Authorizing `identity:create_grant(project_id=701d8b3f08cf4ba6af9f072877249d4c, user_id=54743d61389549f982b874edd65c497e, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:11.605105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e190a4ef-b0d9-4297-a608-c61f5c088878 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:11.618860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1327/2642] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:11 2022] PUT /identity/v3/projects/701d8b3f08cf4ba6af9f072877249d4c/users/54743d61389549f982b874edd65c497e/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:11.682996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d391d427-51bc-4165-9d83-7f4aed41d82e None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.683274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d391d427-51bc-4165-9d83-7f4aed41d82e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.683547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d391d427-51bc-4165-9d83-7f4aed41d82e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.711877 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-d391d427-51bc-4165-9d83-7f4aed41d82e None None] MFA Rules not processed for user `54743d61389549f982b874edd65c497e`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:11.713116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d391d427-51bc-4165-9d83-7f4aed41d82e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:11.781432 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1316/2643] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:11 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:11.793309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e190a4ef-b0d9-4297-a608-c61f5c088878 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:11.795086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e190a4ef-b0d9-4297-a608-c61f5c088878 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:11.804667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:11.855074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e190a4ef-b0d9-4297-a608-c61f5c088878 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:11.858670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793a6f74-d01e-4155-a330-be74cfb3cfa3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:11.859283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793a6f74-d01e-4155-a330-be74cfb3cfa3 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:11.859548 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793a6f74-d01e-4155-a330-be74cfb3cfa3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:11.859820 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793a6f74-d01e-4155-a330-be74cfb3cfa3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:11.869009 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793a6f74-d01e-4155-a330-be74cfb3cfa3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:11.870479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793a6f74-d01e-4155-a330-be74cfb3cfa3 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:11.879295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1328/2644] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:11 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:13.821236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d391d427-51bc-4165-9d83-7f4aed41d82e None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:13.822634 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d391d427-51bc-4165-9d83-7f4aed41d82e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:13.881211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d391d427-51bc-4165-9d83-7f4aed41d82e None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:13.885292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7a6e94-078c-400c-a535-7ce29173b772 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:13.885887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a6e94-078c-400c-a535-7ce29173b772 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:13.886196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a6e94-078c-400c-a535-7ce29173b772 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:13.886477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a6e94-078c-400c-a535-7ce29173b772 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:13.888185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ab7a6e94-078c-400c-a535-7ce29173b772 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:13.949019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7a6e94-078c-400c-a535-7ce29173b772 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:13.950391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7a6e94-078c-400c-a535-7ce29173b772 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:13.960217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1317/2645] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:17.111038 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793a6f74-d01e-4155-a330-be74cfb3cfa3 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:17.112769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-793a6f74-d01e-4155-a330-be74cfb3cfa3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:17.129833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793a6f74-d01e-4155-a330-be74cfb3cfa3 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:17.136726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:17.137361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:17.137637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:17.137906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] PATH_INFO: `/v3/users/6415bc1076204c78ad819ab2836a3652` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:17.139118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6415bc1076204c78ad819ab2836a3652)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:17.141074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:17.199644 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6415bc1076204c78ad819ab2836a3652'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:17.206424 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6415bc1076204c78ad819ab2836a3652'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:17.207084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6415bc1076204c78ad819ab2836a3652'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:17.207696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6415bc1076204c78ad819ab2836a3652'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:17.215110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a9944669-392f-4e63-9be5-aa4b5566c9d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6415bc1076204c78ad819ab2836a3652'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:17.233242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1329/2646] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:17 2022] DELETE /identity/v3/users/6415bc1076204c78ad819ab2836a3652 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:17.423035 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7a6e94-078c-400c-a535-7ce29173b772 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:17.430285 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:17.435704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7a6e94-078c-400c-a535-7ce29173b772 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:17.503309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:17.503925 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:17.504207 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:17.504479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] PATH_INFO: `/v3/projects/6f2f218a688a41ab9e06269c4be3973d` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:17.505570 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f2f218a688a41ab9e06269c4be3973d)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:17.507470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:17.551690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f2f218a688a41ab9e06269c4be3973d', 'request_id': 'req-d4f71b5c-be5d-4c41-a21c-1122153c0a09'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:17.558993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f2f218a688a41ab9e06269c4be3973d', 'request_id': 'req-d4f71b5c-be5d-4c41-a21c-1122153c0a09'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:17.565196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f2f218a688a41ab9e06269c4be3973d', 'request_id': 'req-d4f71b5c-be5d-4c41-a21c-1122153c0a09'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:17.575201 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] The token cache is being invalidate because project 6f2f218a688a41ab9e06269c4be3973d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:17.575486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:17.576100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d4f71b5c-be5d-4c41-a21c-1122153c0a09 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:17.577530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1318/2647] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:17 2022] DELETE /identity/v3/projects/6f2f218a688a41ab9e06269c4be3973d => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:17.582523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef72d20-ea27-4a7f-8d69-11152981a519 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:17.582809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef72d20-ea27-4a7f-8d69-11152981a519 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:17.583113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef72d20-ea27-4a7f-8d69-11152981a519 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:17.603740 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6ef72d20-ea27-4a7f-8d69-11152981a519 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:17.604941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6ef72d20-ea27-4a7f-8d69-11152981a519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:17.662134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1330/2648] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:17.671985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c05abda-3281-45d9-85d2-8a95488c598d None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:17.672269 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c05abda-3281-45d9-85d2-8a95488c598d None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:17.672541 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c05abda-3281-45d9-85d2-8a95488c598d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:17.692181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9c05abda-3281-45d9-85d2-8a95488c598d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:17.693366 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9c05abda-3281-45d9-85d2-8a95488c598d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:17.700876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1319/2649] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:17.704990 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef72d20-ea27-4a7f-8d69-11152981a519 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:17.714405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef72d20-ea27-4a7f-8d69-11152981a519 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:17.718810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8f405d-bfb1-4e99-aed8-fccdc01ae072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:17.719405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8f405d-bfb1-4e99-aed8-fccdc01ae072 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:17.719863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8f405d-bfb1-4e99-aed8-fccdc01ae072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:17.720134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8f405d-bfb1-4e99-aed8-fccdc01ae072 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:17.720758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8f405d-bfb1-4e99-aed8-fccdc01ae072 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:01:17.721053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8f405d-bfb1-4e99-aed8-fccdc01ae072 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:17.722643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8f405d-bfb1-4e99-aed8-fccdc01ae072 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:17.734260 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1331/2650] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:01:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:01:17.738462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c05abda-3281-45d9-85d2-8a95488c598d None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:17.748157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c05abda-3281-45d9-85d2-8a95488c598d None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:17.752614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ee8e8f-0a57-4e13-bdc6-65ff70c25532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:17.753189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ee8e8f-0a57-4e13-bdc6-65ff70c25532 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:17.753461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ee8e8f-0a57-4e13-bdc6-65ff70c25532 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:17.753728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ee8e8f-0a57-4e13-bdc6-65ff70c25532 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:17.754375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ee8e8f-0a57-4e13-bdc6-65ff70c25532 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:17.756225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ee8e8f-0a57-4e13-bdc6-65ff70c25532 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:17.785747 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1320/2651] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:01:17 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:17.790472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8f405d-bfb1-4e99-aed8-fccdc01ae072 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:17.800795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8f405d-bfb1-4e99-aed8-fccdc01ae072 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:17.805433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:17.857085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae736a26-644d-4756-a64c-b93c70e0cee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:17.857685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae736a26-644d-4756-a64c-b93c70e0cee2 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:17.857963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae736a26-644d-4756-a64c-b93c70e0cee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:17.858262 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae736a26-644d-4756-a64c-b93c70e0cee2 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:17.858872 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae736a26-644d-4756-a64c-b93c70e0cee2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:17.860795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae736a26-644d-4756-a64c-b93c70e0cee2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:17.881521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1332/2652] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:01:17 2022] POST /identity/v3/users => generated 292 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:01:17.885639 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ee8e8f-0a57-4e13-bdc6-65ff70c25532 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:17.895685 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ee8e8f-0a57-4e13-bdc6-65ff70c25532 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:17.900753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f86ecc2-f792-4dbe-bad0-026324a2a549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:17.901344 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f86ecc2-f792-4dbe-bad0-026324a2a549 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:17.901624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f86ecc2-f792-4dbe-bad0-026324a2a549 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:17.901893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f86ecc2-f792-4dbe-bad0-026324a2a549 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:17.902565 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f86ecc2-f792-4dbe-bad0-026324a2a549 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:17.904188 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f86ecc2-f792-4dbe-bad0-026324a2a549 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:17.913781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1321/2653] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:17.918173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae736a26-644d-4756-a64c-b93c70e0cee2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:17.927923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae736a26-644d-4756-a64c-b93c70e0cee2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:17.932675 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3b6ab6-95d6-49a5-bba3-26456529ca29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:17.933255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b6ab6-95d6-49a5-bba3-26456529ca29 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:17.933527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b6ab6-95d6-49a5-bba3-26456529ca29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:17.933792 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3b6ab6-95d6-49a5-bba3-26456529ca29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:17.934459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3b6ab6-95d6-49a5-bba3-26456529ca29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:17.936034 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d3b6ab6-95d6-49a5-bba3-26456529ca29 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:17.946359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1333/2654] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:17.950572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f86ecc2-f792-4dbe-bad0-026324a2a549 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:17.960114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f86ecc2-f792-4dbe-bad0-026324a2a549 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:17.965615 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd76d8-8d28-47b1-aef3-0597b91c9529 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:17.966294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd76d8-8d28-47b1-aef3-0597b91c9529 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:17.966572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd76d8-8d28-47b1-aef3-0597b91c9529 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:17.966844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd76d8-8d28-47b1-aef3-0597b91c9529 admin admin] PATH_INFO: `/v3/projects/29870f31b41846cc923a28ebfaf368db/users/1750f2cc69c0499ebade616da183507b/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:17.983463 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbd76d8-8d28-47b1-aef3-0597b91c9529 admin admin] RBAC: Authorizing `identity:create_grant(project_id=29870f31b41846cc923a28ebfaf368db, user_id=1750f2cc69c0499ebade616da183507b, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:17.986709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbd76d8-8d28-47b1-aef3-0597b91c9529 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:18.001696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1322/2655] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:17 2022] PUT /identity/v3/projects/29870f31b41846cc923a28ebfaf368db/users/1750f2cc69c0499ebade616da183507b/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:18.006572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108376c1-64eb-4202-b0d4-64e9126be194 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:18.006856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108376c1-64eb-4202-b0d4-64e9126be194 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:18.007127 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108376c1-64eb-4202-b0d4-64e9126be194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:18.026049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-108376c1-64eb-4202-b0d4-64e9126be194 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:18.027281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-108376c1-64eb-4202-b0d4-64e9126be194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:18.087311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1334/2656] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:18.097052 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd76d8-8d28-47b1-aef3-0597b91c9529 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:18.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5dbd76d8-8d28-47b1-aef3-0597b91c9529 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:18.134643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:18.159969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd76d8-8d28-47b1-aef3-0597b91c9529 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:18.163638 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5658952c-e9ba-4b17-960e-34aca89ee46a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:18.164254 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5658952c-e9ba-4b17-960e-34aca89ee46a service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:18.164528 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5658952c-e9ba-4b17-960e-34aca89ee46a service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:18.164794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5658952c-e9ba-4b17-960e-34aca89ee46a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:18.173424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5658952c-e9ba-4b17-960e-34aca89ee46a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:18.174973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5658952c-e9ba-4b17-960e-34aca89ee46a service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:18.184176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1323/2657] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:22.109393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108376c1-64eb-4202-b0d4-64e9126be194 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:22.110945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-108376c1-64eb-4202-b0d4-64e9126be194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:22.124387 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108376c1-64eb-4202-b0d4-64e9126be194 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:22.128459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1412310-f362-41ba-bc13-479da9b14b2f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:22.129079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1412310-f362-41ba-bc13-479da9b14b2f service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:22.129356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1412310-f362-41ba-bc13-479da9b14b2f service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:22.129808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1412310-f362-41ba-bc13-479da9b14b2f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:22.131995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a1412310-f362-41ba-bc13-479da9b14b2f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:22.221765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1412310-f362-41ba-bc13-479da9b14b2f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:22.223252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1412310-f362-41ba-bc13-479da9b14b2f service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:22.237413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1335/2658] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:22 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:22.450362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e3b2ea-d2c2-4c3e-9350-49c84fe1dcbc None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:22.450867 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e3b2ea-d2c2-4c3e-9350-49c84fe1dcbc None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:22.450967 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e3b2ea-d2c2-4c3e-9350-49c84fe1dcbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:22.481082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-07e3b2ea-d2c2-4c3e-9350-49c84fe1dcbc None None] MFA Rules not processed for user `1750f2cc69c0499ebade616da183507b`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:22.482274 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-07e3b2ea-d2c2-4c3e-9350-49c84fe1dcbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:22.558209 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1324/2659] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:22.572525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1412310-f362-41ba-bc13-479da9b14b2f service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:22.573916 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a1412310-f362-41ba-bc13-479da9b14b2f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:22.604679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:22.653054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1412310-f362-41ba-bc13-479da9b14b2f service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:22.658411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a4d411-7e68-425e-ae2d-0f07ebb3f1e7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:22.659014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a4d411-7e68-425e-ae2d-0f07ebb3f1e7 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:22.659293 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a4d411-7e68-425e-ae2d-0f07ebb3f1e7 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:22.659562 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a4d411-7e68-425e-ae2d-0f07ebb3f1e7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:22.668065 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a4d411-7e68-425e-ae2d-0f07ebb3f1e7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:22.669643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a4d411-7e68-425e-ae2d-0f07ebb3f1e7 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:22.678773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1336/2660] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:22 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:23.329526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e3b2ea-d2c2-4c3e-9350-49c84fe1dcbc None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:23.362594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e3b2ea-d2c2-4c3e-9350-49c84fe1dcbc None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:23.375707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c595a9c-18c9-4c90-abd5-80ff9087447e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:23.378288 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c595a9c-18c9-4c90-abd5-80ff9087447e service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:23.378662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c595a9c-18c9-4c90-abd5-80ff9087447e service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:23.378992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c595a9c-18c9-4c90-abd5-80ff9087447e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:23.384578 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8c595a9c-18c9-4c90-abd5-80ff9087447e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:23.410412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c595a9c-18c9-4c90-abd5-80ff9087447e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:23.410412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c595a9c-18c9-4c90-abd5-80ff9087447e service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:23.425943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1325/2661] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:24.453359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bffc601-c5e2-4aea-a3f2-bf09c3b53810 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:24.453834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bffc601-c5e2-4aea-a3f2-bf09c3b53810 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:24.454091 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bffc601-c5e2-4aea-a3f2-bf09c3b53810 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:24.474847 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4bffc601-c5e2-4aea-a3f2-bf09c3b53810 None None] MFA Rules not processed for user `1750f2cc69c0499ebade616da183507b`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:24.476377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4bffc601-c5e2-4aea-a3f2-bf09c3b53810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:24.483403 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1337/2662] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:24.847718 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c595a9c-18c9-4c90-abd5-80ff9087447e service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:24.861715 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c595a9c-18c9-4c90-abd5-80ff9087447e service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:24.874460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1ca11d-b70d-45af-bb8c-f6761757f66e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:24.875314 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1ca11d-b70d-45af-bb8c-f6761757f66e service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:24.876512 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1ca11d-b70d-45af-bb8c-f6761757f66e service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:24.876940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1ca11d-b70d-45af-bb8c-f6761757f66e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:24.878904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4a1ca11d-b70d-45af-bb8c-f6761757f66e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:24.902704 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1ca11d-b70d-45af-bb8c-f6761757f66e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:24.904352 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1ca11d-b70d-45af-bb8c-f6761757f66e service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:24.918424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1326/2663] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:24 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:29.986053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bffc601-c5e2-4aea-a3f2-bf09c3b53810 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:30.000520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bffc601-c5e2-4aea-a3f2-bf09c3b53810 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:30.004974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47af140-3c39-47d2-baa9-d1e918c7076c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:30.005956 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47af140-3c39-47d2-baa9-d1e918c7076c service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:30.006550 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47af140-3c39-47d2-baa9-d1e918c7076c service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:30.007062 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47af140-3c39-47d2-baa9-d1e918c7076c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:30.009382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b47af140-3c39-47d2-baa9-d1e918c7076c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:30.025710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47af140-3c39-47d2-baa9-d1e918c7076c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:30.029734 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47af140-3c39-47d2-baa9-d1e918c7076c service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:30.063758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1338/2664] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:34.912409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=56 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:35.645603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1ca11d-b70d-45af-bb8c-f6761757f66e service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:35.659518 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1ca11d-b70d-45af-bb8c-f6761757f66e service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:35.663950 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293dd699-9c0f-4e90-af1b-0b8516c4ea3e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:35.664845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293dd699-9c0f-4e90-af1b-0b8516c4ea3e service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:35.665303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293dd699-9c0f-4e90-af1b-0b8516c4ea3e service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:35.665731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293dd699-9c0f-4e90-af1b-0b8516c4ea3e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:35.667954 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-293dd699-9c0f-4e90-af1b-0b8516c4ea3e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:35.682341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293dd699-9c0f-4e90-af1b-0b8516c4ea3e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:35.684062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293dd699-9c0f-4e90-af1b-0b8516c4ea3e service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:35.697136 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1327/2665] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 01:01:38.211915 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47af140-3c39-47d2-baa9-d1e918c7076c service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:38.213575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-b47af140-3c39-47d2-baa9-d1e918c7076c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:38.231758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47af140-3c39-47d2-baa9-d1e918c7076c service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:38.237703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:38.237956 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:38.238303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:38.238580 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] PATH_INFO: `/v3/users/bfee7d93eeff4e728abea067c0c42814` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:38.240066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] RBAC: Authorizing `identity:delete_user(user_id=bfee7d93eeff4e728abea067c0c42814)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:38.244439 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:38.327063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bfee7d93eeff4e728abea067c0c42814'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:38.338931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bfee7d93eeff4e728abea067c0c42814'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:38.339785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bfee7d93eeff4e728abea067c0c42814'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:38.340980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bfee7d93eeff4e728abea067c0c42814'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:38.352143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bfee7d93eeff4e728abea067c0c42814'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:38.386623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1339/2666] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:38 2022] DELETE /identity/v3/users/bfee7d93eeff4e728abea067c0c42814 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:38.867763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293dd699-9c0f-4e90-af1b-0b8516c4ea3e service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:38.879933 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293dd699-9c0f-4e90-af1b-0b8516c4ea3e service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:38.966160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:38.967315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:38.967802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:38.968760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] PATH_INFO: `/v3/projects/302dd84b0b90485691447badb4536c56` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:38.971255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] RBAC: Authorizing `identity:delete_project(project_id=302dd84b0b90485691447badb4536c56)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:38.974341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:39.081069 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '302dd84b0b90485691447badb4536c56', 'request_id': 'req-5ad281b4-0e82-45af-9002-75d2b8144940'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.094974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '302dd84b0b90485691447badb4536c56', 'request_id': 'req-5ad281b4-0e82-45af-9002-75d2b8144940'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.112287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '302dd84b0b90485691447badb4536c56', 'request_id': 'req-5ad281b4-0e82-45af-9002-75d2b8144940'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.130717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] The token cache is being invalidate because project 302dd84b0b90485691447badb4536c56 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:39.131150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.131974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.134816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1328/2667] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:38 2022] DELETE /identity/v3/projects/302dd84b0b90485691447badb4536c56 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:39.137899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:39.141998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:39.168141 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7333d605-d813-458d-bc36-957750cf2bca admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:39.321531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:39.323134 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:39.323581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:39.323845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] PATH_INFO: `/v3/users/b0f9f74f9f3b48e894bd6c3aea35aa3b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:39.325104 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0f9f74f9f3b48e894bd6c3aea35aa3b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:39.327341 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:39.332541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:39.422296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0f9f74f9f3b48e894bd6c3aea35aa3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.431822 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0f9f74f9f3b48e894bd6c3aea35aa3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.442205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0f9f74f9f3b48e894bd6c3aea35aa3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.443203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0f9f74f9f3b48e894bd6c3aea35aa3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.460890 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0f9f74f9f3b48e894bd6c3aea35aa3b'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.506155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1340/2668] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:39 2022] DELETE /identity/v3/users/b0f9f74f9f3b48e894bd6c3aea35aa3b => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:39.765622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:39.768324 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:39.791282 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad281b4-0e82-45af-9002-75d2b8144940 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:39.864839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:39.866412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:39.871447 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:39.871973 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:39.872428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:39.872716 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] PATH_INFO: `/v3/users/a8894fd1044241ee8ba84927b6e8326a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:39.873975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8894fd1044241ee8ba84927b6e8326a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:39.876211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:39.885765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081de927-bb5c-4778-8f0f-5206323460cb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:39.891853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:39.892569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:39.892848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:39.893117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] PATH_INFO: `/v3/projects/0e6bb29020b04fc2b21ca50a53e3b59f` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:39.895057 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e6bb29020b04fc2b21ca50a53e3b59f)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:39.897962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:39.928956 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:39.961637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e6bb29020b04fc2b21ca50a53e3b59f', 'request_id': 'req-03d36dc5-63fc-4f38-9d5d-d7945b17492d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.970596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8894fd1044241ee8ba84927b6e8326a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.971379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8894fd1044241ee8ba84927b6e8326a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.975472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e6bb29020b04fc2b21ca50a53e3b59f', 'request_id': 'req-03d36dc5-63fc-4f38-9d5d-d7945b17492d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.980086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8894fd1044241ee8ba84927b6e8326a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.980984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8894fd1044241ee8ba84927b6e8326a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.982681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e6bb29020b04fc2b21ca50a53e3b59f', 'request_id': 'req-03d36dc5-63fc-4f38-9d5d-d7945b17492d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:39.992293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-d5a853f3-9588-493c-8c7c-2f325ab5dc97 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8894fd1044241ee8ba84927b6e8326a'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:40.010142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] The token cache is being invalidate because project 0e6bb29020b04fc2b21ca50a53e3b59f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:40.010396 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:40.011960 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:40.030892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1341/2669] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:39 2022] DELETE /identity/v3/projects/0e6bb29020b04fc2b21ca50a53e3b59f => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:40.039875 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1329/2670] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:39 2022] DELETE /identity/v3/users/a8894fd1044241ee8ba84927b6e8326a => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:40.402645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:40.405242 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:40.490118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d36dc5-63fc-4f38-9d5d-d7945b17492d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:40.501357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:40.551377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:40.552004 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:40.552279 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:40.552547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] PATH_INFO: `/v3/projects/ff4aef28dfc444ee914b56c64b2edb77` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:40.553676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff4aef28dfc444ee914b56c64b2edb77)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:40.555601 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:40.605158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff4aef28dfc444ee914b56c64b2edb77', 'request_id': 'req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:40.613223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff4aef28dfc444ee914b56c64b2edb77', 'request_id': 'req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:40.619824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff4aef28dfc444ee914b56c64b2edb77', 'request_id': 'req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:40.630569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] The token cache is being invalidate because project ff4aef28dfc444ee914b56c64b2edb77 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:40.630854 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:40.632180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-258eb8b4-2d09-45ec-98aa-b4746a8ef2ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:40.633752 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1342/2671] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:40 2022] DELETE /identity/v3/projects/ff4aef28dfc444ee914b56c64b2edb77 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:40.639654 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad62b06-832e-4ae1-bec3-7080b9ff5bc6 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:40.639934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad62b06-832e-4ae1-bec3-7080b9ff5bc6 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:40.640200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad62b06-832e-4ae1-bec3-7080b9ff5bc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:40.661106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9ad62b06-832e-4ae1-bec3-7080b9ff5bc6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:40.662382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9ad62b06-832e-4ae1-bec3-7080b9ff5bc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:40.728427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1330/2672] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:40.736370 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4f77c7-10cb-4f5c-a222-39ebdc1a0605 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:40.736516 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4f77c7-10cb-4f5c-a222-39ebdc1a0605 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:40.736662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4f77c7-10cb-4f5c-a222-39ebdc1a0605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:40.757253 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-5d4f77c7-10cb-4f5c-a222-39ebdc1a0605 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:40.758470 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5d4f77c7-10cb-4f5c-a222-39ebdc1a0605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:40.766416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1343/2673] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:40.771229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad62b06-832e-4ae1-bec3-7080b9ff5bc6 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:40.782346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad62b06-832e-4ae1-bec3-7080b9ff5bc6 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:40.787014 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea1d35b-96c8-417f-84de-94841f6825b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:40.788163 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea1d35b-96c8-417f-84de-94841f6825b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:40.788444 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea1d35b-96c8-417f-84de-94841f6825b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:40.788717 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea1d35b-96c8-417f-84de-94841f6825b5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:40.789389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea1d35b-96c8-417f-84de-94841f6825b5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:01:40.789696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea1d35b-96c8-417f-84de-94841f6825b5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:40.791340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea1d35b-96c8-417f-84de-94841f6825b5 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:40.808737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1331/2674] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:01:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:01:40.809790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4f77c7-10cb-4f5c-a222-39ebdc1a0605 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:40.821547 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4f77c7-10cb-4f5c-a222-39ebdc1a0605 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:40.832382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa66bb9e-3441-49b7-b629-38e6c7b59a5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:40.838299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66bb9e-3441-49b7-b629-38e6c7b59a5e admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:40.838579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66bb9e-3441-49b7-b629-38e6c7b59a5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:40.838850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa66bb9e-3441-49b7-b629-38e6c7b59a5e admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:40.839519 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa66bb9e-3441-49b7-b629-38e6c7b59a5e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:40.841454 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa66bb9e-3441-49b7-b629-38e6c7b59a5e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:40.881184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1344/2675] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:01:40 2022] POST /identity/v3/projects => generated 391 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:40.882636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea1d35b-96c8-417f-84de-94841f6825b5 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:40.894942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea1d35b-96c8-417f-84de-94841f6825b5 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:40.962553 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782afb46-64d1-4665-a1c1-4a302f8a9aff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:40.963158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782afb46-64d1-4665-a1c1-4a302f8a9aff admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:40.963436 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782afb46-64d1-4665-a1c1-4a302f8a9aff admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:40.963710 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782afb46-64d1-4665-a1c1-4a302f8a9aff admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:40.964337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782afb46-64d1-4665-a1c1-4a302f8a9aff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:40.966266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782afb46-64d1-4665-a1c1-4a302f8a9aff admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:40.989232 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1332/2676] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:01:40 2022] POST /identity/v3/users => generated 296 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:40.994247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa66bb9e-3441-49b7-b629-38e6c7b59a5e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.004623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa66bb9e-3441-49b7-b629-38e6c7b59a5e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.009205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b88be75-6e27-48e9-8159-134102d27080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.009793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b88be75-6e27-48e9-8159-134102d27080 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.010083 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b88be75-6e27-48e9-8159-134102d27080 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.010363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b88be75-6e27-48e9-8159-134102d27080 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.010991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b88be75-6e27-48e9-8159-134102d27080 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.012670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b88be75-6e27-48e9-8159-134102d27080 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.023356 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1345/2677] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:41.027766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782afb46-64d1-4665-a1c1-4a302f8a9aff admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.039969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782afb46-64d1-4665-a1c1-4a302f8a9aff admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.044774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acab3570-5d8a-492a-8903-b95b8eb4499b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.045390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acab3570-5d8a-492a-8903-b95b8eb4499b admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.045667 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acab3570-5d8a-492a-8903-b95b8eb4499b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.045939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acab3570-5d8a-492a-8903-b95b8eb4499b admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.046617 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acab3570-5d8a-492a-8903-b95b8eb4499b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.048202 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acab3570-5d8a-492a-8903-b95b8eb4499b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.060063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1333/2678] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:41.064238 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b88be75-6e27-48e9-8159-134102d27080 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.075101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b88be75-6e27-48e9-8159-134102d27080 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.079736 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0619e82c-2475-46ce-b629-f8522fd847b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.080377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0619e82c-2475-46ce-b629-f8522fd847b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.080658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0619e82c-2475-46ce-b629-f8522fd847b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.080937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0619e82c-2475-46ce-b629-f8522fd847b9 admin admin] PATH_INFO: `/v3/projects/8efab3ef9e8d474f8ff2955e21b3288c/users/b7279050fc4f44a088720d002be46025/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.095597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0619e82c-2475-46ce-b629-f8522fd847b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8efab3ef9e8d474f8ff2955e21b3288c, user_id=b7279050fc4f44a088720d002be46025, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.098275 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0619e82c-2475-46ce-b629-f8522fd847b9 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.110521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1346/2679] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:41 2022] PUT /identity/v3/projects/8efab3ef9e8d474f8ff2955e21b3288c/users/b7279050fc4f44a088720d002be46025/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 01:01:41.121526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599b4a7-9284-4236-b03c-bb30290a8a9e None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.121812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599b4a7-9284-4236-b03c-bb30290a8a9e None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.122104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599b4a7-9284-4236-b03c-bb30290a8a9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.129111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:41.151063 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-2599b4a7-9284-4236-b03c-bb30290a8a9e None None] MFA Rules not processed for user `b7279050fc4f44a088720d002be46025`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:41.152245 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-2599b4a7-9284-4236-b03c-bb30290a8a9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:41.213491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1334/2680] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:41 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:41.218080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0619e82c-2475-46ce-b629-f8522fd847b9 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.259742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:41.279323 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0619e82c-2475-46ce-b629-f8522fd847b9 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.283858 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f1aca3-0d96-4be3-9477-2fab5b6b58d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.284450 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f1aca3-0d96-4be3-9477-2fab5b6b58d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.284722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f1aca3-0d96-4be3-9477-2fab5b6b58d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.284989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f1aca3-0d96-4be3-9477-2fab5b6b58d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.285618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f1aca3-0d96-4be3-9477-2fab5b6b58d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.287524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f1aca3-0d96-4be3-9477-2fab5b6b58d4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.313162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1347/2681] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:01:41 2022] POST /identity/v3/projects => generated 393 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:41.317301 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2599b4a7-9284-4236-b03c-bb30290a8a9e None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.328386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2599b4a7-9284-4236-b03c-bb30290a8a9e None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.382859 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bf9c98-1f1f-446e-b7c4-234ffa4a4bde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.383459 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bf9c98-1f1f-446e-b7c4-234ffa4a4bde admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.383733 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bf9c98-1f1f-446e-b7c4-234ffa4a4bde admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.384009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bf9c98-1f1f-446e-b7c4-234ffa4a4bde admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.384605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bf9c98-1f1f-446e-b7c4-234ffa4a4bde admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.386545 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bf9c98-1f1f-446e-b7c4-234ffa4a4bde admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.406783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1335/2682] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:01:41 2022] POST /identity/v3/users => generated 303 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:41.410826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f1aca3-0d96-4be3-9477-2fab5b6b58d4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.421830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f1aca3-0d96-4be3-9477-2fab5b6b58d4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.426393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587f7708-86e9-4c07-b75e-6ceb3c16d8d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.426976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f7708-86e9-4c07-b75e-6ceb3c16d8d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.427248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f7708-86e9-4c07-b75e-6ceb3c16d8d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.427517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587f7708-86e9-4c07-b75e-6ceb3c16d8d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.428143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587f7708-86e9-4c07-b75e-6ceb3c16d8d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.429812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587f7708-86e9-4c07-b75e-6ceb3c16d8d7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.440408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1348/2683] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:41.445030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bf9c98-1f1f-446e-b7c4-234ffa4a4bde admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.454623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bf9c98-1f1f-446e-b7c4-234ffa4a4bde admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.459077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6be00a0-c428-421b-9cbb-b564d6703b0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.459799 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6be00a0-c428-421b-9cbb-b564d6703b0c admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.460074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6be00a0-c428-421b-9cbb-b564d6703b0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.460341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6be00a0-c428-421b-9cbb-b564d6703b0c admin admin] PATH_INFO: `/v3/projects/7de6ec97763e4542b9d65bd37937787e/users/3f5f242b09aa424ea0857738ed420dc1/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.474695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6be00a0-c428-421b-9cbb-b564d6703b0c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7de6ec97763e4542b9d65bd37937787e, user_id=3f5f242b09aa424ea0857738ed420dc1, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.477843 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6be00a0-c428-421b-9cbb-b564d6703b0c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.490568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1336/2684] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:41 2022] PUT /identity/v3/projects/7de6ec97763e4542b9d65bd37937787e/users/3f5f242b09aa424ea0857738ed420dc1/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:41.494514 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587f7708-86e9-4c07-b75e-6ceb3c16d8d7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.504790 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587f7708-86e9-4c07-b75e-6ceb3c16d8d7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.559610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b7501f-e514-4623-b202-18861dfdef76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.560201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b7501f-e514-4623-b202-18861dfdef76 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.560474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b7501f-e514-4623-b202-18861dfdef76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.560741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b7501f-e514-4623-b202-18861dfdef76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.561361 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b7501f-e514-4623-b202-18861dfdef76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.562968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b7501f-e514-4623-b202-18861dfdef76 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.572986 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1349/2685] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:41.577015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6be00a0-c428-421b-9cbb-b564d6703b0c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.587505 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6be00a0-c428-421b-9cbb-b564d6703b0c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.591946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1380e46-aa77-41ae-bc79-21a8b748252c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.592580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1380e46-aa77-41ae-bc79-21a8b748252c admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.592855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1380e46-aa77-41ae-bc79-21a8b748252c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.593124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1380e46-aa77-41ae-bc79-21a8b748252c admin admin] PATH_INFO: `/v3/projects/7de6ec97763e4542b9d65bd37937787e/users/3f5f242b09aa424ea0857738ed420dc1/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.595019 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1380e46-aa77-41ae-bc79-21a8b748252c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7de6ec97763e4542b9d65bd37937787e, user_id=3f5f242b09aa424ea0857738ed420dc1, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.597575 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1380e46-aa77-41ae-bc79-21a8b748252c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.607479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1337/2686] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:41 2022] PUT /identity/v3/projects/7de6ec97763e4542b9d65bd37937787e/users/3f5f242b09aa424ea0857738ed420dc1/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:41.618984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca793e-660e-49dc-a808-1d07e559aded None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.619268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca793e-660e-49dc-a808-1d07e559aded None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.619542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca793e-660e-49dc-a808-1d07e559aded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.647835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e8ca793e-660e-49dc-a808-1d07e559aded None None] MFA Rules not processed for user `3f5f242b09aa424ea0857738ed420dc1`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:41.649018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e8ca793e-660e-49dc-a808-1d07e559aded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:41.714400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1350/2687] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:41 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:41.725598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1380e46-aa77-41ae-bc79-21a8b748252c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.727266 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f1380e46-aa77-41ae-bc79-21a8b748252c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:41.773789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:41.786738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1380e46-aa77-41ae-bc79-21a8b748252c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.791032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4a1070-f3e6-4e67-8fd8-55097e71f8e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.791666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4a1070-f3e6-4e67-8fd8-55097e71f8e9 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.791943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4a1070-f3e6-4e67-8fd8-55097e71f8e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.792213 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4a1070-f3e6-4e67-8fd8-55097e71f8e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.801465 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4a1070-f3e6-4e67-8fd8-55097e71f8e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.803026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4a1070-f3e6-4e67-8fd8-55097e71f8e9 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.812333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1338/2688] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:41 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:41.863987 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca793e-660e-49dc-a808-1d07e559aded None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:41.866164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e8ca793e-660e-49dc-a808-1d07e559aded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:41.877879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca793e-660e-49dc-a808-1d07e559aded None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:41.881481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41a510a-c3a9-4a8b-9131-c66b73d91bd8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:41.882100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41a510a-c3a9-4a8b-9131-c66b73d91bd8 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:41.882380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41a510a-c3a9-4a8b-9131-c66b73d91bd8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:41.882650 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41a510a-c3a9-4a8b-9131-c66b73d91bd8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:41.915826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:41.941879 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41a510a-c3a9-4a8b-9131-c66b73d91bd8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:41.943338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41a510a-c3a9-4a8b-9131-c66b73d91bd8 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:41.952829 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1351/2689] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:41 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:45.181695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4a1070-f3e6-4e67-8fd8-55097e71f8e9 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:45.241158 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4a1070-f3e6-4e67-8fd8-55097e71f8e9 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:45.245663 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:45.246293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:45.246573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:45.246848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] PATH_INFO: `/v3/users/b7279050fc4f44a088720d002be46025` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:45.247947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7279050fc4f44a088720d002be46025)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:45.249832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:45.294653 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7279050fc4f44a088720d002be46025'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:45.295299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7279050fc4f44a088720d002be46025'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:45.302561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7279050fc4f44a088720d002be46025'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:45.303647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7279050fc4f44a088720d002be46025'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:45.310598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7279050fc4f44a088720d002be46025'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:45.327455 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1339/2690] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:45 2022] DELETE /identity/v3/users/b7279050fc4f44a088720d002be46025 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:45.331963 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4ebccb-31c9-46bf-b8e9-eabb77ee0a2c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:45.332414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4ebccb-31c9-46bf-b8e9-eabb77ee0a2c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:45.332529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4ebccb-31c9-46bf-b8e9-eabb77ee0a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:45.351523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-3b4ebccb-31c9-46bf-b8e9-eabb77ee0a2c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:45.352703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3b4ebccb-31c9-46bf-b8e9-eabb77ee0a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:45.409966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1352/2691] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:45.419850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:45.421230 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:45.483199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f59106-5c3d-45ab-9b8e-b375a7ef64e0 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:45.486958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c152ba-2267-40de-88a9-9a0c0fbba34e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:45.487584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c152ba-2267-40de-88a9-9a0c0fbba34e service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:45.487856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c152ba-2267-40de-88a9-9a0c0fbba34e service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:45.488125 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c152ba-2267-40de-88a9-9a0c0fbba34e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:45.497889 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c152ba-2267-40de-88a9-9a0c0fbba34e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:45.499350 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c152ba-2267-40de-88a9-9a0c0fbba34e service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:45.509175 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1340/2692] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:45.781079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4ebccb-31c9-46bf-b8e9-eabb77ee0a2c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:45.782552 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3b4ebccb-31c9-46bf-b8e9-eabb77ee0a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:45.796677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4ebccb-31c9-46bf-b8e9-eabb77ee0a2c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:45.801408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:45.802761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:45.803721 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:45.803999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] PATH_INFO: `/v3/projects/8efab3ef9e8d474f8ff2955e21b3288c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:45.805209 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] RBAC: Authorizing `identity:delete_project(project_id=8efab3ef9e8d474f8ff2955e21b3288c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:45.807138 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:45.855319 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8efab3ef9e8d474f8ff2955e21b3288c', 'request_id': 'req-8b81b330-b2d9-4d52-bff8-8a556a47badb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:45.861521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8efab3ef9e8d474f8ff2955e21b3288c', 'request_id': 'req-8b81b330-b2d9-4d52-bff8-8a556a47badb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:45.868596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8efab3ef9e8d474f8ff2955e21b3288c', 'request_id': 'req-8b81b330-b2d9-4d52-bff8-8a556a47badb'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:45.879809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] The token cache is being invalidate because project 8efab3ef9e8d474f8ff2955e21b3288c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:45.880114 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:45.880809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:45.882436 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1353/2693] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:45 2022] DELETE /identity/v3/projects/8efab3ef9e8d474f8ff2955e21b3288c => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:45.886569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c152ba-2267-40de-88a9-9a0c0fbba34e service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:45.896219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c152ba-2267-40de-88a9-9a0c0fbba34e service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:45.902017 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:45.960134 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:45.960751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:45.961026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:45.961295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] PATH_INFO: `/v3/users/3f5f242b09aa424ea0857738ed420dc1` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:45.962407 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f5f242b09aa424ea0857738ed420dc1)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:45.964390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:46.009881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f5f242b09aa424ea0857738ed420dc1'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:46.010563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f5f242b09aa424ea0857738ed420dc1'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:46.017261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f5f242b09aa424ea0857738ed420dc1'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:46.017898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f5f242b09aa424ea0857738ed420dc1'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:46.025573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-85e650e8-4caf-4071-b5fb-ff179c92d02b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f5f242b09aa424ea0857738ed420dc1'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:46.046048 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1341/2694] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:45 2022] DELETE /identity/v3/users/3f5f242b09aa424ea0857738ed420dc1 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:46.305177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:46.306671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:46.319993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b81b330-b2d9-4d52-bff8-8a556a47badb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:46.370326 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:46.376926 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:46.377841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:46.378154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:46.378438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] PATH_INFO: `/v3/projects/7de6ec97763e4542b9d65bd37937787e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:46.379758 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] RBAC: Authorizing `identity:delete_project(project_id=7de6ec97763e4542b9d65bd37937787e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:46.382636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:46.427195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7de6ec97763e4542b9d65bd37937787e', 'request_id': 'req-be30fffc-48b5-4a6c-992a-d32593a0260f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:46.434730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7de6ec97763e4542b9d65bd37937787e', 'request_id': 'req-be30fffc-48b5-4a6c-992a-d32593a0260f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:46.440934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7de6ec97763e4542b9d65bd37937787e', 'request_id': 'req-be30fffc-48b5-4a6c-992a-d32593a0260f'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:46.451651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] The token cache is being invalidate because project 7de6ec97763e4542b9d65bd37937787e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:46.451938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:46.452561 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-be30fffc-48b5-4a6c-992a-d32593a0260f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:46.453998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1354/2695] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:46 2022] DELETE /identity/v3/projects/7de6ec97763e4542b9d65bd37937787e => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:46.459556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8838a45-b2c0-4693-ba15-b89f6ca1f217 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:46.460046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8838a45-b2c0-4693-ba15-b89f6ca1f217 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:46.460155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8838a45-b2c0-4693-ba15-b89f6ca1f217 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:46.479116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-d8838a45-b2c0-4693-ba15-b89f6ca1f217 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:46.480294 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d8838a45-b2c0-4693-ba15-b89f6ca1f217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:46.538560 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1342/2696] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:46.546366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76922c5d-a398-4b8b-8f09-0d99dc1dfd30 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:46.546648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76922c5d-a398-4b8b-8f09-0d99dc1dfd30 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:46.546924 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76922c5d-a398-4b8b-8f09-0d99dc1dfd30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:46.566646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-76922c5d-a398-4b8b-8f09-0d99dc1dfd30 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:46.567842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-76922c5d-a398-4b8b-8f09-0d99dc1dfd30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:46.576363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1355/2697] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:46.580545 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8838a45-b2c0-4693-ba15-b89f6ca1f217 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:46.590086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8838a45-b2c0-4693-ba15-b89f6ca1f217 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:46.594581 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4c2679-c8ea-42f1-a218-95e03dd37d5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:46.595173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c2679-c8ea-42f1-a218-95e03dd37d5f admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:46.595450 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c2679-c8ea-42f1-a218-95e03dd37d5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:46.595723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c2679-c8ea-42f1-a218-95e03dd37d5f admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:46.596360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4c2679-c8ea-42f1-a218-95e03dd37d5f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:01:46.596898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4c2679-c8ea-42f1-a218-95e03dd37d5f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:46.598506 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4c2679-c8ea-42f1-a218-95e03dd37d5f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:46.610353 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1343/2698] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:01:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:01:46.614728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76922c5d-a398-4b8b-8f09-0d99dc1dfd30 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:46.624309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76922c5d-a398-4b8b-8f09-0d99dc1dfd30 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:46.628918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fac47d-e7bf-49de-96d2-c4f21e2c76ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:46.629498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fac47d-e7bf-49de-96d2-c4f21e2c76ea admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:46.629773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fac47d-e7bf-49de-96d2-c4f21e2c76ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:46.630042 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fac47d-e7bf-49de-96d2-c4f21e2c76ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:46.630696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fac47d-e7bf-49de-96d2-c4f21e2c76ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:46.632586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fac47d-e7bf-49de-96d2-c4f21e2c76ea admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:46.659603 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1356/2699] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:01:46 2022] POST /identity/v3/projects => generated 387 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:46.664229 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4c2679-c8ea-42f1-a218-95e03dd37d5f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:46.673976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4c2679-c8ea-42f1-a218-95e03dd37d5f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:46.686190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:46.736091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce23ba0-6ce0-4fb8-b573-cdb1bf951ee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:46.736697 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce23ba0-6ce0-4fb8-b573-cdb1bf951ee4 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:46.736976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce23ba0-6ce0-4fb8-b573-cdb1bf951ee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:46.737461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce23ba0-6ce0-4fb8-b573-cdb1bf951ee4 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:46.737889 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce23ba0-6ce0-4fb8-b573-cdb1bf951ee4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:46.740086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce23ba0-6ce0-4fb8-b573-cdb1bf951ee4 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:46.762563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1344/2700] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:01:46 2022] POST /identity/v3/users => generated 294 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:46.766774 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fac47d-e7bf-49de-96d2-c4f21e2c76ea admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:46.785914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fac47d-e7bf-49de-96d2-c4f21e2c76ea admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:46.791110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adb0523-e61f-4968-9392-c8b952b242ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:46.791699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adb0523-e61f-4968-9392-c8b952b242ae admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:46.791972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adb0523-e61f-4968-9392-c8b952b242ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:46.792237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adb0523-e61f-4968-9392-c8b952b242ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:46.792865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adb0523-e61f-4968-9392-c8b952b242ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:46.794476 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adb0523-e61f-4968-9392-c8b952b242ae admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:46.805218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1357/2701] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:46 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:46.809786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce23ba0-6ce0-4fb8-b573-cdb1bf951ee4 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:46.820929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce23ba0-6ce0-4fb8-b573-cdb1bf951ee4 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:46.825790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6add49b4-3a51-4e25-b916-462df13daeb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:46.826468 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add49b4-3a51-4e25-b916-462df13daeb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:46.826735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add49b4-3a51-4e25-b916-462df13daeb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:46.827009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add49b4-3a51-4e25-b916-462df13daeb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:46.827666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6add49b4-3a51-4e25-b916-462df13daeb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:46.829340 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6add49b4-3a51-4e25-b916-462df13daeb1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:46.841395 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1345/2702] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:46 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:46.845830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adb0523-e61f-4968-9392-c8b952b242ae admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:46.857161 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adb0523-e61f-4968-9392-c8b952b242ae admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:46.861948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4720177e-48f6-4647-84eb-9024024d3501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:46.862660 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720177e-48f6-4647-84eb-9024024d3501 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:46.862935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720177e-48f6-4647-84eb-9024024d3501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:46.863205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720177e-48f6-4647-84eb-9024024d3501 admin admin] PATH_INFO: `/v3/projects/11c0de440c0c4506909eb207e33cf06f/users/51f7363105de495ba83fcb90e91b82e9/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:46.880421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4720177e-48f6-4647-84eb-9024024d3501 admin admin] RBAC: Authorizing `identity:create_grant(project_id=11c0de440c0c4506909eb207e33cf06f, user_id=51f7363105de495ba83fcb90e91b82e9, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:46.883830 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4720177e-48f6-4647-84eb-9024024d3501 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:46.903352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1358/2703] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:46 2022] PUT /identity/v3/projects/11c0de440c0c4506909eb207e33cf06f/users/51f7363105de495ba83fcb90e91b82e9/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:46.916751 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610bc0a-98ba-45e2-8e60-39d4de5840a8 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:46.917040 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610bc0a-98ba-45e2-8e60-39d4de5840a8 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:46.917317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610bc0a-98ba-45e2-8e60-39d4de5840a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:46.948149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-6610bc0a-98ba-45e2-8e60-39d4de5840a8 None None] MFA Rules not processed for user `51f7363105de495ba83fcb90e91b82e9`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:46.949411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6610bc0a-98ba-45e2-8e60-39d4de5840a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:47.016686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1346/2704] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:46 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:47.021113 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4720177e-48f6-4647-84eb-9024024d3501 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:47.084453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4720177e-48f6-4647-84eb-9024024d3501 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:47.089128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29044259-5027-417c-82ad-b86ddc681de3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:47.089733 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29044259-5027-417c-82ad-b86ddc681de3 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:47.090008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29044259-5027-417c-82ad-b86ddc681de3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:47.090324 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29044259-5027-417c-82ad-b86ddc681de3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:47.090930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29044259-5027-417c-82ad-b86ddc681de3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:47.092824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29044259-5027-417c-82ad-b86ddc681de3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:47.120148 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1359/2705] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:01:47 2022] POST /identity/v3/projects => generated 387 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:47.124259 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610bc0a-98ba-45e2-8e60-39d4de5840a8 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:47.135384 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610bc0a-98ba-45e2-8e60-39d4de5840a8 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:47.190934 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176db170-88d9-4e8e-879e-94e34e89dc4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:47.191540 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176db170-88d9-4e8e-879e-94e34e89dc4a admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:47.191904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176db170-88d9-4e8e-879e-94e34e89dc4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:47.192178 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176db170-88d9-4e8e-879e-94e34e89dc4a admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:47.192817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176db170-88d9-4e8e-879e-94e34e89dc4a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:47.195442 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176db170-88d9-4e8e-879e-94e34e89dc4a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:47.216227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1347/2706] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:01:47 2022] POST /identity/v3/users => generated 300 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:47.220195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29044259-5027-417c-82ad-b86ddc681de3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:47.230545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29044259-5027-417c-82ad-b86ddc681de3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:47.235164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5c9ee1-c8f9-4b45-b228-45d4225bcfb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:47.235767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5c9ee1-c8f9-4b45-b228-45d4225bcfb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:47.236044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5c9ee1-c8f9-4b45-b228-45d4225bcfb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:47.236311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5c9ee1-c8f9-4b45-b228-45d4225bcfb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:47.236941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5c9ee1-c8f9-4b45-b228-45d4225bcfb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:47.238574 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5c9ee1-c8f9-4b45-b228-45d4225bcfb3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:47.241698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:47.252645 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1360/2707] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 01:01:47.257203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176db170-88d9-4e8e-879e-94e34e89dc4a admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:47.267047 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176db170-88d9-4e8e-879e-94e34e89dc4a admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:47.272488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72377d59-40c3-447d-912b-34054e8eff01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:47.273152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72377d59-40c3-447d-912b-34054e8eff01 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:47.273438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72377d59-40c3-447d-912b-34054e8eff01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:47.273712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72377d59-40c3-447d-912b-34054e8eff01 admin admin] PATH_INFO: `/v3/projects/63501ae2f1794807ab1ea947c4fdbc85/users/74fe00699c244a7d8b35913f8788493d/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:47.285566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:47.291043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72377d59-40c3-447d-912b-34054e8eff01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63501ae2f1794807ab1ea947c4fdbc85, user_id=74fe00699c244a7d8b35913f8788493d, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:47.294020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72377d59-40c3-447d-912b-34054e8eff01 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:47.306927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1348/2708] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:47 2022] PUT /identity/v3/projects/63501ae2f1794807ab1ea947c4fdbc85/users/74fe00699c244a7d8b35913f8788493d/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:47.311017 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5c9ee1-c8f9-4b45-b228-45d4225bcfb3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:47.321162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5c9ee1-c8f9-4b45-b228-45d4225bcfb3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:47.377084 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c1fab5-cc30-482f-80af-d668653df0ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:47.377684 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c1fab5-cc30-482f-80af-d668653df0ff admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:47.377954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c1fab5-cc30-482f-80af-d668653df0ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:47.378249 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c1fab5-cc30-482f-80af-d668653df0ff admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:47.378880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c1fab5-cc30-482f-80af-d668653df0ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:47.380438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c1fab5-cc30-482f-80af-d668653df0ff admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:47.390761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1361/2709] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:47 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:47.395007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72377d59-40c3-447d-912b-34054e8eff01 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:47.405257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72377d59-40c3-447d-912b-34054e8eff01 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:47.409862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71051ab1-9420-46c3-9603-cdb6f616c17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:47.410545 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71051ab1-9420-46c3-9603-cdb6f616c17c admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:47.410825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71051ab1-9420-46c3-9603-cdb6f616c17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:47.411100 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71051ab1-9420-46c3-9603-cdb6f616c17c admin admin] PATH_INFO: `/v3/projects/63501ae2f1794807ab1ea947c4fdbc85/users/74fe00699c244a7d8b35913f8788493d/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:47.413491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71051ab1-9420-46c3-9603-cdb6f616c17c admin admin] RBAC: Authorizing `identity:create_grant(project_id=63501ae2f1794807ab1ea947c4fdbc85, user_id=74fe00699c244a7d8b35913f8788493d, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:47.416129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71051ab1-9420-46c3-9603-cdb6f616c17c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:47.426408 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1349/2710] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:47 2022] PUT /identity/v3/projects/63501ae2f1794807ab1ea947c4fdbc85/users/74fe00699c244a7d8b35913f8788493d/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:47.437320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31b6753-650a-418b-92c8-eba76ce9062a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:47.437607 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31b6753-650a-418b-92c8-eba76ce9062a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:47.437886 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31b6753-650a-418b-92c8-eba76ce9062a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:47.465445 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c31b6753-650a-418b-92c8-eba76ce9062a None None] MFA Rules not processed for user `74fe00699c244a7d8b35913f8788493d`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:47.466785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c31b6753-650a-418b-92c8-eba76ce9062a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:47.533231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1362/2711] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:47 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:47.548148 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71051ab1-9420-46c3-9603-cdb6f616c17c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:47.549846 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-71051ab1-9420-46c3-9603-cdb6f616c17c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:47.608085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71051ab1-9420-46c3-9603-cdb6f616c17c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:47.611670 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4434f2-bc91-4d09-bafe-02f1d24b1b19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:47.612261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4434f2-bc91-4d09-bafe-02f1d24b1b19 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:47.612541 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4434f2-bc91-4d09-bafe-02f1d24b1b19 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:47.612810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4434f2-bc91-4d09-bafe-02f1d24b1b19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:47.622394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4434f2-bc91-4d09-bafe-02f1d24b1b19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:47.623789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4434f2-bc91-4d09-bafe-02f1d24b1b19 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:47.633453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1350/2712] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:47 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:47.766461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31b6753-650a-418b-92c8-eba76ce9062a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:47.828798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31b6753-650a-418b-92c8-eba76ce9062a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:47.833233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:47.833849 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:47.834651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:47.834921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] PATH_INFO: `/v3/users/51f7363105de495ba83fcb90e91b82e9` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:47.836019 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51f7363105de495ba83fcb90e91b82e9)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:47.837911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:47.861408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:47.892378 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51f7363105de495ba83fcb90e91b82e9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:47.900812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51f7363105de495ba83fcb90e91b82e9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:47.901529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51f7363105de495ba83fcb90e91b82e9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:47.903150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51f7363105de495ba83fcb90e91b82e9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:47.911742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51f7363105de495ba83fcb90e91b82e9'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:47.934842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1363/2713] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:47 2022] DELETE /identity/v3/users/51f7363105de495ba83fcb90e91b82e9 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:47.939328 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6306f88-9037-4b9e-b858-1da6da8472c9 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:47.939614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6306f88-9037-4b9e-b858-1da6da8472c9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:47.939888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6306f88-9037-4b9e-b858-1da6da8472c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:47.958866 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-c6306f88-9037-4b9e-b858-1da6da8472c9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:47.960693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c6306f88-9037-4b9e-b858-1da6da8472c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:48.017345 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1351/2714] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:48.027673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:48.029146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:48.086952 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8bf89a-75da-4005-9cad-18dcc45e87e5 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:48.090728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f9423b-b27c-449e-a38c-3220bbb94dd8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:48.091332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f9423b-b27c-449e-a38c-3220bbb94dd8 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:48.091610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f9423b-b27c-449e-a38c-3220bbb94dd8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:48.091891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f9423b-b27c-449e-a38c-3220bbb94dd8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:48.101922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f9423b-b27c-449e-a38c-3220bbb94dd8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:48.103604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f9423b-b27c-449e-a38c-3220bbb94dd8 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:48.113846 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1364/2715] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:48.389264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6306f88-9037-4b9e-b858-1da6da8472c9 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:48.390706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c6306f88-9037-4b9e-b858-1da6da8472c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:48.402402 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6306f88-9037-4b9e-b858-1da6da8472c9 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:48.407066 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:48.407842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:48.408051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:48.408211 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] PATH_INFO: `/v3/projects/11c0de440c0c4506909eb207e33cf06f` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:48.410417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=11c0de440c0c4506909eb207e33cf06f)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:48.413376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:48.459979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '11c0de440c0c4506909eb207e33cf06f', 'request_id': 'req-7be7ecd8-ae09-4143-854a-9d359a852e6c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:48.466656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '11c0de440c0c4506909eb207e33cf06f', 'request_id': 'req-7be7ecd8-ae09-4143-854a-9d359a852e6c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:48.472789 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '11c0de440c0c4506909eb207e33cf06f', 'request_id': 'req-7be7ecd8-ae09-4143-854a-9d359a852e6c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:48.483403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] The token cache is being invalidate because project 11c0de440c0c4506909eb207e33cf06f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:48.483690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:48.484303 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:48.485744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1352/2716] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:48 2022] DELETE /identity/v3/projects/11c0de440c0c4506909eb207e33cf06f => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:48.489971 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f9423b-b27c-449e-a38c-3220bbb94dd8 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:48.499995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f9423b-b27c-449e-a38c-3220bbb94dd8 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:48.555753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888737b9-242b-428c-a192-9516be677228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:48.556303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888737b9-242b-428c-a192-9516be677228 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:48.556579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888737b9-242b-428c-a192-9516be677228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:48.556880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888737b9-242b-428c-a192-9516be677228 admin admin] PATH_INFO: `/v3/users/74fe00699c244a7d8b35913f8788493d` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:48.558005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888737b9-242b-428c-a192-9516be677228 admin admin] RBAC: Authorizing `identity:delete_user(user_id=74fe00699c244a7d8b35913f8788493d)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:48.559933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888737b9-242b-428c-a192-9516be677228 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:48.606202 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:48.609247 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-888737b9-242b-428c-a192-9516be677228 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74fe00699c244a7d8b35913f8788493d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:48.618784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-888737b9-242b-428c-a192-9516be677228 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74fe00699c244a7d8b35913f8788493d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:48.619434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-888737b9-242b-428c-a192-9516be677228 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74fe00699c244a7d8b35913f8788493d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:48.620028 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-888737b9-242b-428c-a192-9516be677228 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74fe00699c244a7d8b35913f8788493d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:48.627255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-888737b9-242b-428c-a192-9516be677228 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74fe00699c244a7d8b35913f8788493d'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:48.644100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1365/2717] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:48 2022] DELETE /identity/v3/users/74fe00699c244a7d8b35913f8788493d => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:48.908092 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:48.912721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:48.922461 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:48.930893 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be7ecd8-ae09-4143-854a-9d359a852e6c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:48.948920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888737b9-242b-428c-a192-9516be677228 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:48.950389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-888737b9-242b-428c-a192-9516be677228 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:48.988601 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:48.989219 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:48.989759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:48.990025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] PATH_INFO: `/v3/projects/63501ae2f1794807ab1ea947c4fdbc85` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:48.991217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] RBAC: Authorizing `identity:delete_project(project_id=63501ae2f1794807ab1ea947c4fdbc85)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:48.993603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.015082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888737b9-242b-428c-a192-9516be677228 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.019773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.020368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.020649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.020920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] PATH_INFO: `/v3/users/54743d61389549f982b874edd65c497e` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.022116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=54743d61389549f982b874edd65c497e)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.024043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.046498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63501ae2f1794807ab1ea947c4fdbc85', 'request_id': 'req-de7cc57c-42e0-4d61-9fa3-5a780b10742f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.054495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63501ae2f1794807ab1ea947c4fdbc85', 'request_id': 'req-de7cc57c-42e0-4d61-9fa3-5a780b10742f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.060980 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63501ae2f1794807ab1ea947c4fdbc85', 'request_id': 'req-de7cc57c-42e0-4d61-9fa3-5a780b10742f'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.071720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] The token cache is being invalidate because project 63501ae2f1794807ab1ea947c4fdbc85 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:49.072091 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.072731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-de7cc57c-42e0-4d61-9fa3-5a780b10742f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.073960 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54743d61389549f982b874edd65c497e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.074309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1353/2718] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:48 2022] DELETE /identity/v3/projects/63501ae2f1794807ab1ea947c4fdbc85 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:49.078970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea29f4b-4a28-49c7-a38a-5d0fe01fe6ec None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.079307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea29f4b-4a28-49c7-a38a-5d0fe01fe6ec None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.079584 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea29f4b-4a28-49c7-a38a-5d0fe01fe6ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.081463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54743d61389549f982b874edd65c497e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.082287 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54743d61389549f982b874edd65c497e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.082884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54743d61389549f982b874edd65c497e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.091229 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-4c6ab9cf-3306-4301-aafb-b0f9c7837f99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54743d61389549f982b874edd65c497e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.100622 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-9ea29f4b-4a28-49c7-a38a-5d0fe01fe6ec None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:49.101826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9ea29f4b-4a28-49c7-a38a-5d0fe01fe6ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:49.111352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1366/2719] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:48 2022] DELETE /identity/v3/users/54743d61389549f982b874edd65c497e => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 01:01:49.116596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7be474b-4743-4fb7-adae-c6f66d5ada6e None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.116882 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7be474b-4743-4fb7-adae-c6f66d5ada6e None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.117177 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7be474b-4743-4fb7-adae-c6f66d5ada6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.135377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a7be474b-4743-4fb7-adae-c6f66d5ada6e None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:49.136571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a7be474b-4743-4fb7-adae-c6f66d5ada6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:49.161024 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1354/2720] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:01:49.169257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02be4fdd-b0ee-40d8-bb3e-99d80b30ce84 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.169546 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02be4fdd-b0ee-40d8-bb3e-99d80b30ce84 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.169820 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02be4fdd-b0ee-40d8-bb3e-99d80b30ce84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.175539 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:49.189969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-02be4fdd-b0ee-40d8-bb3e-99d80b30ce84 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:49.191198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-02be4fdd-b0ee-40d8-bb3e-99d80b30ce84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:49.203741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1367/2721] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:01:49.214203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7be474b-4743-4fb7-adae-c6f66d5ada6e None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.215670 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a7be474b-4743-4fb7-adae-c6f66d5ada6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:49.222821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:49.255400 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1355/2722] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:01:49.260180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02be4fdd-b0ee-40d8-bb3e-99d80b30ce84 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.269204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02be4fdd-b0ee-40d8-bb3e-99d80b30ce84 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.272904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7be474b-4743-4fb7-adae-c6f66d5ada6e None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.274145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21529d5d-d7fa-4d9e-892a-5111cced008f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.274741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21529d5d-d7fa-4d9e-892a-5111cced008f admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.275020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21529d5d-d7fa-4d9e-892a-5111cced008f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.275291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21529d5d-d7fa-4d9e-892a-5111cced008f admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.275929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21529d5d-d7fa-4d9e-892a-5111cced008f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:01:49.276258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21529d5d-d7fa-4d9e-892a-5111cced008f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.276594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4fba80-ebb6-4246-8af5-3021acd40d57 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.277186 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4fba80-ebb6-4246-8af5-3021acd40d57 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.277458 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4fba80-ebb6-4246-8af5-3021acd40d57 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.277725 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4fba80-ebb6-4246-8af5-3021acd40d57 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.277894 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21529d5d-d7fa-4d9e-892a-5111cced008f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.286702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4fba80-ebb6-4246-8af5-3021acd40d57 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.288150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4fba80-ebb6-4246-8af5-3021acd40d57 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.292810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1356/2723] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:01:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 01:01:49.293798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21529d5d-d7fa-4d9e-892a-5111cced008f admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.298240 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1368/2724] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:49.303831 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21529d5d-d7fa-4d9e-892a-5111cced008f admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.308469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd91816-c084-4ef2-9d7b-9fe9afb409fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.309058 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd91816-c084-4ef2-9d7b-9fe9afb409fe admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.309337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd91816-c084-4ef2-9d7b-9fe9afb409fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.309616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd91816-c084-4ef2-9d7b-9fe9afb409fe admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.310252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd91816-c084-4ef2-9d7b-9fe9afb409fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.312120 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd91816-c084-4ef2-9d7b-9fe9afb409fe admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.340120 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1357/2725] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:01:49.344682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4fba80-ebb6-4246-8af5-3021acd40d57 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.354508 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4fba80-ebb6-4246-8af5-3021acd40d57 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.413268 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd78c3-29a8-4e76-a9d1-f6664a6c9f1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.413880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd78c3-29a8-4e76-a9d1-f6664a6c9f1c admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.414197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd78c3-29a8-4e76-a9d1-f6664a6c9f1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.414480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd78c3-29a8-4e76-a9d1-f6664a6c9f1c admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.415100 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd78c3-29a8-4e76-a9d1-f6664a6c9f1c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.417049 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd78c3-29a8-4e76-a9d1-f6664a6c9f1c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.438599 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1369/2726] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/users => generated 293 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:49.442586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd91816-c084-4ef2-9d7b-9fe9afb409fe admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.454526 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd91816-c084-4ef2-9d7b-9fe9afb409fe admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.459223 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542823d1-d291-404b-8d10-c7964edb4e00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.459828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542823d1-d291-404b-8d10-c7964edb4e00 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.460106 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542823d1-d291-404b-8d10-c7964edb4e00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.460373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542823d1-d291-404b-8d10-c7964edb4e00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.461006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542823d1-d291-404b-8d10-c7964edb4e00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.462618 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542823d1-d291-404b-8d10-c7964edb4e00 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.472566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1358/2727] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 01:01:49.476591 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd78c3-29a8-4e76-a9d1-f6664a6c9f1c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.486405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd78c3-29a8-4e76-a9d1-f6664a6c9f1c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.491833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0b614d-1ae7-4b15-b980-00660b761c80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.492428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0b614d-1ae7-4b15-b980-00660b761c80 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.492677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0b614d-1ae7-4b15-b980-00660b761c80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.492946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0b614d-1ae7-4b15-b980-00660b761c80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.493639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0b614d-1ae7-4b15-b980-00660b761c80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.495200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0b614d-1ae7-4b15-b980-00660b761c80 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.513197 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1370/2728] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:49.518379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542823d1-d291-404b-8d10-c7964edb4e00 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.532503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542823d1-d291-404b-8d10-c7964edb4e00 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.537906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8393aad7-7beb-454b-8b34-8733a10673fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.538600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8393aad7-7beb-454b-8b34-8733a10673fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.538891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8393aad7-7beb-454b-8b34-8733a10673fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.539166 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8393aad7-7beb-454b-8b34-8733a10673fe admin admin] PATH_INFO: `/v3/projects/3d7167a55a3843aebfdb59f7d083781b/users/06c95ddb78614feaa14f5c3ef513ab7c/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.554988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8393aad7-7beb-454b-8b34-8733a10673fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d7167a55a3843aebfdb59f7d083781b, user_id=06c95ddb78614feaa14f5c3ef513ab7c, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.557988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8393aad7-7beb-454b-8b34-8733a10673fe admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.579276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1359/2729] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:49 2022] PUT /identity/v3/projects/3d7167a55a3843aebfdb59f7d083781b/users/06c95ddb78614feaa14f5c3ef513ab7c/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:49.580314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0b614d-1ae7-4b15-b980-00660b761c80 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.582122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4f0b614d-1ae7-4b15-b980-00660b761c80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:49.582616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68ae339-f679-44ad-9dfd-9de6a880ed2a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.582986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68ae339-f679-44ad-9dfd-9de6a880ed2a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.583498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68ae339-f679-44ad-9dfd-9de6a880ed2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.598311 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0b614d-1ae7-4b15-b980-00660b761c80 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.618363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a68ae339-f679-44ad-9dfd-9de6a880ed2a None None] MFA Rules not processed for user `06c95ddb78614feaa14f5c3ef513ab7c`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:49.619577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a68ae339-f679-44ad-9dfd-9de6a880ed2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:49.659189 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.659813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.660089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.660359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] PATH_INFO: `/v3/projects/701d8b3f08cf4ba6af9f072877249d4c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.661485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] RBAC: Authorizing `identity:delete_project(project_id=701d8b3f08cf4ba6af9f072877249d4c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.663455 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.665749 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:49.684699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1360/2730] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:49.689099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68ae339-f679-44ad-9dfd-9de6a880ed2a None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.700409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68ae339-f679-44ad-9dfd-9de6a880ed2a None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.712107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '701d8b3f08cf4ba6af9f072877249d4c', 'request_id': 'req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.719981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '701d8b3f08cf4ba6af9f072877249d4c', 'request_id': 'req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.726895 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '701d8b3f08cf4ba6af9f072877249d4c', 'request_id': 'req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.738782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] The token cache is being invalidate because project 701d8b3f08cf4ba6af9f072877249d4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:49.739092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.740014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-0ccdd9ed-fe75-4fb9-979e-06a0a429d898 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:49.741498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1371/2731] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:49 2022] DELETE /identity/v3/projects/701d8b3f08cf4ba6af9f072877249d4c => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:49.746321 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1517851-352c-4860-83fd-d95029361a85 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.746600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1517851-352c-4860-83fd-d95029361a85 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.746874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1517851-352c-4860-83fd-d95029361a85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.747993 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:49.758413 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ab4ad6-2462-4f51-90e3-8e55bd75cccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.759015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ab4ad6-2462-4f51-90e3-8e55bd75cccc admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.759292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ab4ad6-2462-4f51-90e3-8e55bd75cccc admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.759561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ab4ad6-2462-4f51-90e3-8e55bd75cccc admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.760194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ab4ad6-2462-4f51-90e3-8e55bd75cccc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.762071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ab4ad6-2462-4f51-90e3-8e55bd75cccc admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.767156 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c1517851-352c-4860-83fd-d95029361a85 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:49.768779 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c1517851-352c-4860-83fd-d95029361a85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:49.777716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1372/2732] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:01:49.785164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88614764-7eae-4c98-a866-fd89ff595c9c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.785447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88614764-7eae-4c98-a866-fd89ff595c9c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.785713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88614764-7eae-4c98-a866-fd89ff595c9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.790124 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1361/2733] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/projects => generated 385 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:01:49.794088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ab4ad6-2462-4f51-90e3-8e55bd75cccc admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.796115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-88ab4ad6-2462-4f51-90e3-8e55bd75cccc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:49.805703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-88614764-7eae-4c98-a866-fd89ff595c9c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:49.807086 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-88614764-7eae-4c98-a866-fd89ff595c9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:49.859271 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ab4ad6-2462-4f51-90e3-8e55bd75cccc admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.863786 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47429620-fa9c-498c-b0c2-e9c023f9fa99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.864626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47429620-fa9c-498c-b0c2-e9c023f9fa99 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.864908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47429620-fa9c-498c-b0c2-e9c023f9fa99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.865176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47429620-fa9c-498c-b0c2-e9c023f9fa99 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.865815 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47429620-fa9c-498c-b0c2-e9c023f9fa99 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.867173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1373/2734] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:01:49.867768 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47429620-fa9c-498c-b0c2-e9c023f9fa99 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.871659 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88614764-7eae-4c98-a866-fd89ff595c9c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.880511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88614764-7eae-4c98-a866-fd89ff595c9c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.886213 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae9775b-bc5a-4a3b-895c-5bcddeba97fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.886832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae9775b-bc5a-4a3b-895c-5bcddeba97fc admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.887095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae9775b-bc5a-4a3b-895c-5bcddeba97fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.887363 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae9775b-bc5a-4a3b-895c-5bcddeba97fc admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.888010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae9775b-bc5a-4a3b-895c-5bcddeba97fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:01:49.888310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae9775b-bc5a-4a3b-895c-5bcddeba97fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.889892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae9775b-bc5a-4a3b-895c-5bcddeba97fc admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.890104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1362/2735] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/users => generated 299 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:49.894054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47429620-fa9c-498c-b0c2-e9c023f9fa99 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.902567 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1374/2736] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:01:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 02 01:01:49.905154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47429620-fa9c-498c-b0c2-e9c023f9fa99 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.906894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae9775b-bc5a-4a3b-895c-5bcddeba97fc admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.909952 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a301c1-8d7d-4d51-8ad5-69703cc1682a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.910567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a301c1-8d7d-4d51-8ad5-69703cc1682a admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.910840 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a301c1-8d7d-4d51-8ad5-69703cc1682a admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.911105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a301c1-8d7d-4d51-8ad5-69703cc1682a admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.911732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a301c1-8d7d-4d51-8ad5-69703cc1682a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.913339 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12a301c1-8d7d-4d51-8ad5-69703cc1682a admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.916922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae9775b-bc5a-4a3b-895c-5bcddeba97fc admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.921521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211ca850-67a4-478a-93d0-c0b66f8f17ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.922124 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ca850-67a4-478a-93d0-c0b66f8f17ef admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.922404 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ca850-67a4-478a-93d0-c0b66f8f17ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.922888 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ca850-67a4-478a-93d0-c0b66f8f17ef admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.923515 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211ca850-67a4-478a-93d0-c0b66f8f17ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.923826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1363/2737] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 01:01:49.925524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211ca850-67a4-478a-93d0-c0b66f8f17ef admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.927874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a301c1-8d7d-4d51-8ad5-69703cc1682a admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.937430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12a301c1-8d7d-4d51-8ad5-69703cc1682a admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:49.942606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0714a8de-ec4a-4989-95a7-211228a94750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:49.943268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0714a8de-ec4a-4989-95a7-211228a94750 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:49.944120 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0714a8de-ec4a-4989-95a7-211228a94750 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:49.944440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0714a8de-ec4a-4989-95a7-211228a94750 admin admin] PATH_INFO: `/v3/projects/4c4d5059f72b43e9b8fd567b97a11bca/users/4b4b35ad635d4dc6bbcbce4d0cac3f43/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:49.956783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1375/2738] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:01:49.959015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211ca850-67a4-478a-93d0-c0b66f8f17ef admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:49.959892 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0714a8de-ec4a-4989-95a7-211228a94750 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c4d5059f72b43e9b8fd567b97a11bca, user_id=4b4b35ad635d4dc6bbcbce4d0cac3f43, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:49.962595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0714a8de-ec4a-4989-95a7-211228a94750 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:49.976453 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1364/2739] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:49 2022] PUT /identity/v3/projects/4c4d5059f72b43e9b8fd567b97a11bca/users/4b4b35ad635d4dc6bbcbce4d0cac3f43/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 01:01:49.980480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0714a8de-ec4a-4989-95a7-211228a94750 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:50.012264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:50.024222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211ca850-67a4-478a-93d0-c0b66f8f17ef admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:50.042729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0714a8de-ec4a-4989-95a7-211228a94750 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:50.047346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f28100e-f57c-4640-b817-5751aa766be3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:50.047943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28100e-f57c-4640-b817-5751aa766be3 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.048318 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28100e-f57c-4640-b817-5751aa766be3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.048594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28100e-f57c-4640-b817-5751aa766be3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.049221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f28100e-f57c-4640-b817-5751aa766be3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:50.050825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f28100e-f57c-4640-b817-5751aa766be3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:50.061237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1365/2740] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:50.065566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f28100e-f57c-4640-b817-5751aa766be3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:50.076021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f28100e-f57c-4640-b817-5751aa766be3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:50.081097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de5ec09-1c09-4a80-be15-ccf016348498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:50.081753 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de5ec09-1c09-4a80-be15-ccf016348498 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.082028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de5ec09-1c09-4a80-be15-ccf016348498 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.082359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de5ec09-1c09-4a80-be15-ccf016348498 admin admin] PATH_INFO: `/v3/projects/4c4d5059f72b43e9b8fd567b97a11bca/users/4b4b35ad635d4dc6bbcbce4d0cac3f43/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.084970 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de5ec09-1c09-4a80-be15-ccf016348498 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c4d5059f72b43e9b8fd567b97a11bca, user_id=4b4b35ad635d4dc6bbcbce4d0cac3f43, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:50.085252 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f1e129-caca-45ca-a992-77a29def1e82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:50.085884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1e129-caca-45ca-a992-77a29def1e82 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.086188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1e129-caca-45ca-a992-77a29def1e82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.086456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f1e129-caca-45ca-a992-77a29def1e82 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.087099 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f1e129-caca-45ca-a992-77a29def1e82 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:50.087871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de5ec09-1c09-4a80-be15-ccf016348498 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:50.088979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f1e129-caca-45ca-a992-77a29def1e82 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:50.098575 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1366/2741] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:50 2022] PUT /identity/v3/projects/4c4d5059f72b43e9b8fd567b97a11bca/users/4b4b35ad635d4dc6bbcbce4d0cac3f43/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 01:01:50.110076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6704e624-5f4b-463c-a627-dffbe6fcbcd1 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.110391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6704e624-5f4b-463c-a627-dffbe6fcbcd1 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.110731 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6704e624-5f4b-463c-a627-dffbe6fcbcd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.115052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1376/2742] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:01:49 2022] POST /identity/v3/users => generated 287 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:01:50.116149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f1e129-caca-45ca-a992-77a29def1e82 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:50.125606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f1e129-caca-45ca-a992-77a29def1e82 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:50.138410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-6704e624-5f4b-463c-a627-dffbe6fcbcd1 None None] MFA Rules not processed for user `4b4b35ad635d4dc6bbcbce4d0cac3f43`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:50.139606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6704e624-5f4b-463c-a627-dffbe6fcbcd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:50.143514 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:50.184998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e06dc4c-fc15-4ee3-a333-ee7f1bca27c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:50.185597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06dc4c-fc15-4ee3-a333-ee7f1bca27c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.185867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06dc4c-fc15-4ee3-a333-ee7f1bca27c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.186231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e06dc4c-fc15-4ee3-a333-ee7f1bca27c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.186851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e06dc4c-fc15-4ee3-a333-ee7f1bca27c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:50.188428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e06dc4c-fc15-4ee3-a333-ee7f1bca27c4 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:50.197868 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1377/2743] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:50 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:50.202081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e06dc4c-fc15-4ee3-a333-ee7f1bca27c4 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:50.211423 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1367/2744] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:50 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:01:50.212276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e06dc4c-fc15-4ee3-a333-ee7f1bca27c4 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:50.216941 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a30f973-488a-4a5e-aba8-de90e2f61104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:50.217468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a30f973-488a-4a5e-aba8-de90e2f61104 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.217744 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a30f973-488a-4a5e-aba8-de90e2f61104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.218012 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a30f973-488a-4a5e-aba8-de90e2f61104 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.218718 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a30f973-488a-4a5e-aba8-de90e2f61104 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:50.220314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a30f973-488a-4a5e-aba8-de90e2f61104 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:50.222440 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6704e624-5f4b-463c-a627-dffbe6fcbcd1 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:50.223752 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6704e624-5f4b-463c-a627-dffbe6fcbcd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:50.233657 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1378/2745] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 02 01:01:50.234861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a30f973-488a-4a5e-aba8-de90e2f61104 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:50.244502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a30f973-488a-4a5e-aba8-de90e2f61104 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:50.249037 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b41e05-ae72-484e-b4b5-926bcd5e1489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:50.249648 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b41e05-ae72-484e-b4b5-926bcd5e1489 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.249921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b41e05-ae72-484e-b4b5-926bcd5e1489 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.250241 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b41e05-ae72-484e-b4b5-926bcd5e1489 admin admin] PATH_INFO: `/v3/projects/2f2f827bbe1a4ef18a20c4c042e511a9/users/11dcf8843c2144b9b1f8e3ceef137384/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.266123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b41e05-ae72-484e-b4b5-926bcd5e1489 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f2f827bbe1a4ef18a20c4c042e511a9, user_id=11dcf8843c2144b9b1f8e3ceef137384, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:50.269006 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b41e05-ae72-484e-b4b5-926bcd5e1489 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:50.279979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6704e624-5f4b-463c-a627-dffbe6fcbcd1 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:50.282442 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1379/2746] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:50 2022] PUT /identity/v3/projects/2f2f827bbe1a4ef18a20c4c042e511a9/users/11dcf8843c2144b9b1f8e3ceef137384/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 01:01:50.293802 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c1ce05-1d7c-44fb-a520-b4a3cc3c79b4 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.294111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c1ce05-1d7c-44fb-a520-b4a3cc3c79b4 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.294392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c1ce05-1d7c-44fb-a520-b4a3cc3c79b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.321798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-c5c1ce05-1d7c-44fb-a520-b4a3cc3c79b4 None None] MFA Rules not processed for user `11dcf8843c2144b9b1f8e3ceef137384`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:50.323204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c5c1ce05-1d7c-44fb-a520-b4a3cc3c79b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:50.328886 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8429406f-c563-4000-a791-3becafb03c8b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:50.329485 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8429406f-c563-4000-a791-3becafb03c8b service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.329760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8429406f-c563-4000-a791-3becafb03c8b service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.330076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8429406f-c563-4000-a791-3becafb03c8b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.331979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8429406f-c563-4000-a791-3becafb03c8b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:50.361473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:50.387984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1380/2747] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:50 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:01:50.393487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8429406f-c563-4000-a791-3becafb03c8b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:50.395155 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8429406f-c563-4000-a791-3becafb03c8b service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:50.400207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c1ce05-1d7c-44fb-a520-b4a3cc3c79b4 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:50.401742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c5c1ce05-1d7c-44fb-a520-b4a3cc3c79b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:50.404852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1368/2748] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:50 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 01:01:50.413641 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c1ce05-1d7c-44fb-a520-b4a3cc3c79b4 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:50.417259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d3ad9f-652d-4bf1-9c3b-212e87e8dd2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:50.417848 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d3ad9f-652d-4bf1-9c3b-212e87e8dd2f service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.418151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d3ad9f-652d-4bf1-9c3b-212e87e8dd2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.418425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d3ad9f-652d-4bf1-9c3b-212e87e8dd2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.426582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d3ad9f-652d-4bf1-9c3b-212e87e8dd2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:50.428001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d3ad9f-652d-4bf1-9c3b-212e87e8dd2f service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:50.436570 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1381/2749] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:50 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 01:01:50.461788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8429406f-c563-4000-a791-3becafb03c8b service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:50.473896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:50.526330 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8429406f-c563-4000-a791-3becafb03c8b service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:50.530841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:50.531460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.531739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.532006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] PATH_INFO: `/v3/users/11dcf8843c2144b9b1f8e3ceef137384` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.533159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] RBAC: Authorizing `identity:delete_user(user_id=11dcf8843c2144b9b1f8e3ceef137384)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:50.535127 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:50.582951 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11dcf8843c2144b9b1f8e3ceef137384'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:50.583626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11dcf8843c2144b9b1f8e3ceef137384'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:50.591647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11dcf8843c2144b9b1f8e3ceef137384'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:50.592309 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11dcf8843c2144b9b1f8e3ceef137384'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:50.599369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11dcf8843c2144b9b1f8e3ceef137384'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:50.618742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1369/2750] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:50 2022] DELETE /identity/v3/users/11dcf8843c2144b9b1f8e3ceef137384 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:50.623372 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6b74d8-1fde-4e8d-ba18-c00dc1823040 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.623859 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6b74d8-1fde-4e8d-ba18-c00dc1823040 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.623979 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6b74d8-1fde-4e8d-ba18-c00dc1823040 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.643459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fa6b74d8-1fde-4e8d-ba18-c00dc1823040 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:50.644673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fa6b74d8-1fde-4e8d-ba18-c00dc1823040 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:50.706506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1382/2751] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:01:50.720315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:50.721742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:50.783720 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747f1062-d420-4eb8-8794-23478071432c admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:50.787331 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276dffb6-f61f-4c4b-9307-0f740a4b9cf8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:50.787931 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276dffb6-f61f-4c4b-9307-0f740a4b9cf8 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:50.788204 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276dffb6-f61f-4c4b-9307-0f740a4b9cf8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:50.788471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276dffb6-f61f-4c4b-9307-0f740a4b9cf8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:50.798932 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276dffb6-f61f-4c4b-9307-0f740a4b9cf8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:50.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276dffb6-f61f-4c4b-9307-0f740a4b9cf8 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:50.810863 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1370/2752] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:51.073816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6b74d8-1fde-4e8d-ba18-c00dc1823040 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:51.075380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fa6b74d8-1fde-4e8d-ba18-c00dc1823040 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:51.089465 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6b74d8-1fde-4e8d-ba18-c00dc1823040 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:51.094828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:51.095541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.095892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.096296 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] PATH_INFO: `/v3/projects/2f2f827bbe1a4ef18a20c4c042e511a9` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.097386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] RBAC: Authorizing `identity:delete_project(project_id=2f2f827bbe1a4ef18a20c4c042e511a9)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:51.099460 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:51.147901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2f2f827bbe1a4ef18a20c4c042e511a9', 'request_id': 'req-a127ced5-e43e-4f00-a863-b5d678ceb27a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:51.154970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2f2f827bbe1a4ef18a20c4c042e511a9', 'request_id': 'req-a127ced5-e43e-4f00-a863-b5d678ceb27a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:51.160930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2f2f827bbe1a4ef18a20c4c042e511a9', 'request_id': 'req-a127ced5-e43e-4f00-a863-b5d678ceb27a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:51.171092 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] The token cache is being invalidate because project 2f2f827bbe1a4ef18a20c4c042e511a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:01:51.171364 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:51.172107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a127ced5-e43e-4f00-a863-b5d678ceb27a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:51.173894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1383/2753] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:01:51 2022] DELETE /identity/v3/projects/2f2f827bbe1a4ef18a20c4c042e511a9 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:51.178961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9c8878-db6c-414d-b8ee-d1b4644c176c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.179429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9c8878-db6c-414d-b8ee-d1b4644c176c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.179554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9c8878-db6c-414d-b8ee-d1b4644c176c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.199563 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-cf9c8878-db6c-414d-b8ee-d1b4644c176c None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:51.200756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-cf9c8878-db6c-414d-b8ee-d1b4644c176c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:51.230763 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:51.262205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1371/2754] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:01:51.269825 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbfd6d8-e224-466f-bf52-e7405f14f8e5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.270173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbfd6d8-e224-466f-bf52-e7405f14f8e5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.270449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbfd6d8-e224-466f-bf52-e7405f14f8e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.290544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-fcbfd6d8-e224-466f-bf52-e7405f14f8e5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:51.291748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-fcbfd6d8-e224-466f-bf52-e7405f14f8e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:51.299523 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1384/2755] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:51.303671 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9c8878-db6c-414d-b8ee-d1b4644c176c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:51.313238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9c8878-db6c-414d-b8ee-d1b4644c176c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:51.317728 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed480b7-6e93-4222-aea1-699c1934348e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:51.318858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed480b7-6e93-4222-aea1-699c1934348e admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.319135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed480b7-6e93-4222-aea1-699c1934348e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.319403 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed480b7-6e93-4222-aea1-699c1934348e admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.320038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed480b7-6e93-4222-aea1-699c1934348e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:01:51.320338 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed480b7-6e93-4222-aea1-699c1934348e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:51.321911 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed480b7-6e93-4222-aea1-699c1934348e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:51.333038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1372/2756] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:01:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:01:51.337891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbfd6d8-e224-466f-bf52-e7405f14f8e5 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:51.346874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbfd6d8-e224-466f-bf52-e7405f14f8e5 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:51.351395 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96fe286-9e42-43a5-b959-e295f021147c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:51.351974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96fe286-9e42-43a5-b959-e295f021147c admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.352248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96fe286-9e42-43a5-b959-e295f021147c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.352521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96fe286-9e42-43a5-b959-e295f021147c admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.353146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96fe286-9e42-43a5-b959-e295f021147c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:51.355070 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96fe286-9e42-43a5-b959-e295f021147c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:51.381724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1385/2757] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:01:51 2022] POST /identity/v3/projects => generated 383 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:01:51.385797 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed480b7-6e93-4222-aea1-699c1934348e admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:51.395415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed480b7-6e93-4222-aea1-699c1934348e admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:51.449067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b922c920-39ff-496a-a94b-67d426c6f081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:51.449586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b922c920-39ff-496a-a94b-67d426c6f081 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.449862 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b922c920-39ff-496a-a94b-67d426c6f081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.450160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b922c920-39ff-496a-a94b-67d426c6f081 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.450776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b922c920-39ff-496a-a94b-67d426c6f081 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:51.452656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b922c920-39ff-496a-a94b-67d426c6f081 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:51.472842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1373/2758] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:01:51 2022] POST /identity/v3/users => generated 292 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:01:51.476887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96fe286-9e42-43a5-b959-e295f021147c admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:51.486995 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96fe286-9e42-43a5-b959-e295f021147c admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:51.492282 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01693800-a4bf-4f3a-abaf-dee325d2511d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:51.492866 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01693800-a4bf-4f3a-abaf-dee325d2511d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.493140 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01693800-a4bf-4f3a-abaf-dee325d2511d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.493409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01693800-a4bf-4f3a-abaf-dee325d2511d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.494088 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01693800-a4bf-4f3a-abaf-dee325d2511d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:51.495674 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01693800-a4bf-4f3a-abaf-dee325d2511d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:51.505317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1386/2759] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:51 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:51.509849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b922c920-39ff-496a-a94b-67d426c6f081 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:51.519539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b922c920-39ff-496a-a94b-67d426c6f081 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:51.523978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45721068-d1ab-441d-8276-d98a8180bb42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:51.524558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45721068-d1ab-441d-8276-d98a8180bb42 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.524833 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45721068-d1ab-441d-8276-d98a8180bb42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.525105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45721068-d1ab-441d-8276-d98a8180bb42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.525732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45721068-d1ab-441d-8276-d98a8180bb42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:51.527334 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45721068-d1ab-441d-8276-d98a8180bb42 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:51.537439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1374/2760] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:01:51 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:01:51.544560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01693800-a4bf-4f3a-abaf-dee325d2511d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:51.551116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01693800-a4bf-4f3a-abaf-dee325d2511d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:51.556144 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ce7d69-2452-4c75-9641-42c60bedccaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:51.556773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ce7d69-2452-4c75-9641-42c60bedccaf admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.557052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ce7d69-2452-4c75-9641-42c60bedccaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.557325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ce7d69-2452-4c75-9641-42c60bedccaf admin admin] PATH_INFO: `/v3/projects/31b72d052142491dbfc1936f4f006715/users/a84c6036067e4ed2817f56d95eaf1715/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.560318 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:51.574782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ce7d69-2452-4c75-9641-42c60bedccaf admin admin] RBAC: Authorizing `identity:create_grant(project_id=31b72d052142491dbfc1936f4f006715, user_id=a84c6036067e4ed2817f56d95eaf1715, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:51.577485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ce7d69-2452-4c75-9641-42c60bedccaf admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:51.591224 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1387/2761] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:01:51 2022] PUT /identity/v3/projects/31b72d052142491dbfc1936f4f006715/users/a84c6036067e4ed2817f56d95eaf1715/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:51.596279 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0ecc2-2595-427f-a8f1-ad61aed074de None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.596561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0ecc2-2595-427f-a8f1-ad61aed074de None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.596857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0ecc2-2595-427f-a8f1-ad61aed074de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.615958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-54b0ecc2-2595-427f-a8f1-ad61aed074de None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:51.617143 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-54b0ecc2-2595-427f-a8f1-ad61aed074de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:51.672632 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1375/2762] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:51.682698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ce7d69-2452-4c75-9641-42c60bedccaf admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:51.684437 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-38ce7d69-2452-4c75-9641-42c60bedccaf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:51.738421 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ce7d69-2452-4c75-9641-42c60bedccaf admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:51.742727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4760ec-5490-4a50-a3e1-16c2fd793692 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:51.743313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4760ec-5490-4a50-a3e1-16c2fd793692 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:51.743586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4760ec-5490-4a50-a3e1-16c2fd793692 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:51.743852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4760ec-5490-4a50-a3e1-16c2fd793692 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:51.752435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4760ec-5490-4a50-a3e1-16c2fd793692 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:51.753842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4760ec-5490-4a50-a3e1-16c2fd793692 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:51.763598 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1388/2763] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:52.409121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88175b4-22eb-4c05-9964-bbc8dc8fb2e7 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:52.409682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88175b4-22eb-4c05-9964-bbc8dc8fb2e7 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:52.409682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88175b4-22eb-4c05-9964-bbc8dc8fb2e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:52.437511 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f88175b4-22eb-4c05-9964-bbc8dc8fb2e7 None None] MFA Rules not processed for user `a84c6036067e4ed2817f56d95eaf1715`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:52.438737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f88175b4-22eb-4c05-9964-bbc8dc8fb2e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:52.466882 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:52.505173 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1376/2764] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:52 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:52.516783 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4760ec-5490-4a50-a3e1-16c2fd793692 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:52.518392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2c4760ec-5490-4a50-a3e1-16c2fd793692 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:52.580035 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4760ec-5490-4a50-a3e1-16c2fd793692 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:52.583516 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13a5ab4-bc10-4cc3-8ee5-d38c8eb3e87d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:52.584658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13a5ab4-bc10-4cc3-8ee5-d38c8eb3e87d service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:52.584933 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13a5ab4-bc10-4cc3-8ee5-d38c8eb3e87d service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:52.585199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13a5ab4-bc10-4cc3-8ee5-d38c8eb3e87d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:52.593669 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13a5ab4-bc10-4cc3-8ee5-d38c8eb3e87d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:52.595118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13a5ab4-bc10-4cc3-8ee5-d38c8eb3e87d service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:52.604189 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1389/2765] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:52 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:53.960452 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b5999c-1761-4897-b93f-459c77d1fcce None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:53.960971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b5999c-1761-4897-b93f-459c77d1fcce None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:53.960971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b5999c-1761-4897-b93f-459c77d1fcce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:53.980790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-95b5999c-1761-4897-b93f-459c77d1fcce None None] MFA Rules not processed for user `a84c6036067e4ed2817f56d95eaf1715`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:53.981985 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-95b5999c-1761-4897-b93f-459c77d1fcce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:53.990181 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1377/2766] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:53 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:54.697123 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13a5ab4-bc10-4cc3-8ee5-d38c8eb3e87d service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:54.698702 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c13a5ab4-bc10-4cc3-8ee5-d38c8eb3e87d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:54.711103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13a5ab4-bc10-4cc3-8ee5-d38c8eb3e87d service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:54.714839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7030c27-7286-4202-abec-730dc8465162 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:54.715413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7030c27-7286-4202-abec-730dc8465162 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:54.715741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7030c27-7286-4202-abec-730dc8465162 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:54.716016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7030c27-7286-4202-abec-730dc8465162 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:54.717706 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e7030c27-7286-4202-abec-730dc8465162 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:54.789729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7030c27-7286-4202-abec-730dc8465162 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:54.792001 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7030c27-7286-4202-abec-730dc8465162 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:54.804384 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1390/2767] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:54 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:01:59.719895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b5999c-1761-4897-b93f-459c77d1fcce None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:59.721299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-95b5999c-1761-4897-b93f-459c77d1fcce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:59.734310 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b5999c-1761-4897-b93f-459c77d1fcce None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:59.738690 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:59.739583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:59.740348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:59.740621 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] PATH_INFO: `/v3/users/06c95ddb78614feaa14f5c3ef513ab7c` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:59.741719 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] RBAC: Authorizing `identity:delete_user(user_id=06c95ddb78614feaa14f5c3ef513ab7c)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:59.743659 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:01:59.791391 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '06c95ddb78614feaa14f5c3ef513ab7c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:59.792104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '06c95ddb78614feaa14f5c3ef513ab7c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:59.799006 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '06c95ddb78614feaa14f5c3ef513ab7c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:59.799627 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '06c95ddb78614feaa14f5c3ef513ab7c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:59.806850 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '06c95ddb78614feaa14f5c3ef513ab7c'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:01:59.824397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1378/2768] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:01:59 2022] DELETE /identity/v3/users/06c95ddb78614feaa14f5c3ef513ab7c => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:01:59.828999 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dbe954-822f-4b6f-b267-638e43750e65 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:59.829524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dbe954-822f-4b6f-b267-638e43750e65 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:59.829524 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73dbe954-822f-4b6f-b267-638e43750e65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:59.848651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-73dbe954-822f-4b6f-b267-638e43750e65 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:01:59.850510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-73dbe954-822f-4b6f-b267-638e43750e65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:59.868902 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:01:59.909766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1391/2769] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:01:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:01:59.921706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:01:59.923112 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:01:59.982462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37094a59-75c7-4008-b3c0-6d9cbf654220 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:01:59.985992 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dc2888-f397-4ba4-8909-5a696e8fedb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:01:59.986625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dc2888-f397-4ba4-8909-5a696e8fedb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:01:59.986905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dc2888-f397-4ba4-8909-5a696e8fedb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:01:59.987176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dc2888-f397-4ba4-8909-5a696e8fedb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:01:59.996474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dc2888-f397-4ba4-8909-5a696e8fedb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:01:59.997898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dc2888-f397-4ba4-8909-5a696e8fedb7 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:00.007221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1379/2770] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:01:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:02:00.266423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dbe954-822f-4b6f-b267-638e43750e65 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:00.267946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-73dbe954-822f-4b6f-b267-638e43750e65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:00.281112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73dbe954-822f-4b6f-b267-638e43750e65 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:00.285742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:00.286398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:00.286676 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:00.286946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] PATH_INFO: `/v3/projects/3d7167a55a3843aebfdb59f7d083781b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:00.288072 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d7167a55a3843aebfdb59f7d083781b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:00.289993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:00.338654 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d7167a55a3843aebfdb59f7d083781b', 'request_id': 'req-39bedf40-a8e3-4516-8a53-21cc6e45590e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.346353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d7167a55a3843aebfdb59f7d083781b', 'request_id': 'req-39bedf40-a8e3-4516-8a53-21cc6e45590e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.353485 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d7167a55a3843aebfdb59f7d083781b', 'request_id': 'req-39bedf40-a8e3-4516-8a53-21cc6e45590e'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.364171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] The token cache is being invalidate because project 3d7167a55a3843aebfdb59f7d083781b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:02:00.364716 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.365939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.367478 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1392/2771] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:02:00 2022] DELETE /identity/v3/projects/3d7167a55a3843aebfdb59f7d083781b => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:02:00.371417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dc2888-f397-4ba4-8909-5a696e8fedb7 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:00.382538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dc2888-f397-4ba4-8909-5a696e8fedb7 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:00.416275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:00.441169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:00.441788 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:00.442075 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:00.442355 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] PATH_INFO: `/v3/users/4b4b35ad635d4dc6bbcbce4d0cac3f43` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:00.444000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b4b35ad635d4dc6bbcbce4d0cac3f43)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:00.445881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:00.493441 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b4b35ad635d4dc6bbcbce4d0cac3f43'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.494203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b4b35ad635d4dc6bbcbce4d0cac3f43'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.501203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b4b35ad635d4dc6bbcbce4d0cac3f43'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.501883 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b4b35ad635d4dc6bbcbce4d0cac3f43'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.509730 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-714f9c58-b6a8-42f5-acf8-125a2d79164f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b4b35ad635d4dc6bbcbce4d0cac3f43'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.526549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1380/2772] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:02:00 2022] DELETE /identity/v3/users/4b4b35ad635d4dc6bbcbce4d0cac3f43 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:02:00.786857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:00.788428 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:00.803254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bedf40-a8e3-4516-8a53-21cc6e45590e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:00.861305 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:00.861948 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:00.862206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:00.862475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] PATH_INFO: `/v3/projects/4c4d5059f72b43e9b8fd567b97a11bca` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:00.863564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c4d5059f72b43e9b8fd567b97a11bca)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:00.866391 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:00.910367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c4d5059f72b43e9b8fd567b97a11bca', 'request_id': 'req-2970c035-b231-4fd5-9b48-9fd83fe60714'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.912473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:00.918532 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c4d5059f72b43e9b8fd567b97a11bca', 'request_id': 'req-2970c035-b231-4fd5-9b48-9fd83fe60714'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.926277 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c4d5059f72b43e9b8fd567b97a11bca', 'request_id': 'req-2970c035-b231-4fd5-9b48-9fd83fe60714'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.937245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] The token cache is being invalidate because project 4c4d5059f72b43e9b8fd567b97a11bca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:02:00.937546 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.938285 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-2970c035-b231-4fd5-9b48-9fd83fe60714 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:00.939728 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1393/2773] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:02:00 2022] DELETE /identity/v3/projects/4c4d5059f72b43e9b8fd567b97a11bca => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:02:00.945845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d577349-baa6-4550-99fe-58b72f862bb8 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:00.946370 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d577349-baa6-4550-99fe-58b72f862bb8 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:00.946471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d577349-baa6-4550-99fe-58b72f862bb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:00.965701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8d577349-baa6-4550-99fe-58b72f862bb8 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:02:00.966920 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8d577349-baa6-4550-99fe-58b72f862bb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:01.024474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1381/2774] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:02:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:02:01.034276 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e3b7d5-750c-46df-a752-92a1038b32b6 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:01.034506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e3b7d5-750c-46df-a752-92a1038b32b6 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:01.034761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e3b7d5-750c-46df-a752-92a1038b32b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:01.056352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a9e3b7d5-750c-46df-a752-92a1038b32b6 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:02:01.057553 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a9e3b7d5-750c-46df-a752-92a1038b32b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:01.065466 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1394/2775] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:02:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:02:01.069913 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d577349-baa6-4550-99fe-58b72f862bb8 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:01.079386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d577349-baa6-4550-99fe-58b72f862bb8 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:01.083785 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feae6cc-9ad1-4bc2-bb2a-336a96205bda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:01.084393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feae6cc-9ad1-4bc2-bb2a-336a96205bda admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:01.084668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feae6cc-9ad1-4bc2-bb2a-336a96205bda admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:01.084936 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feae6cc-9ad1-4bc2-bb2a-336a96205bda admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:01.085572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feae6cc-9ad1-4bc2-bb2a-336a96205bda admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:02:01.085871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feae6cc-9ad1-4bc2-bb2a-336a96205bda admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:01.087498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feae6cc-9ad1-4bc2-bb2a-336a96205bda admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:01.099053 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1382/2776] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:02:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:02:01.103976 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e3b7d5-750c-46df-a752-92a1038b32b6 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:01.113303 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e3b7d5-750c-46df-a752-92a1038b32b6 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:01.118991 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18080924-b840-449d-a45d-9e6816a29666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:01.119580 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18080924-b840-449d-a45d-9e6816a29666 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:01.119857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18080924-b840-449d-a45d-9e6816a29666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:01.120128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18080924-b840-449d-a45d-9e6816a29666 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:01.120748 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18080924-b840-449d-a45d-9e6816a29666 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:01.122679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18080924-b840-449d-a45d-9e6816a29666 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:01.149214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1395/2777] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:02:01 2022] POST /identity/v3/projects => generated 369 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:02:01.153437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feae6cc-9ad1-4bc2-bb2a-336a96205bda admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:01.163554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feae6cc-9ad1-4bc2-bb2a-336a96205bda admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:01.204051 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:01.219462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca0adbb-2267-447e-b18a-0a2926b1f5a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:01.220056 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca0adbb-2267-447e-b18a-0a2926b1f5a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:01.220335 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca0adbb-2267-447e-b18a-0a2926b1f5a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:01.220606 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca0adbb-2267-447e-b18a-0a2926b1f5a8 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:01.221237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca0adbb-2267-447e-b18a-0a2926b1f5a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:01.223168 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca0adbb-2267-447e-b18a-0a2926b1f5a8 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:01.244890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1383/2778] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:02:01 2022] POST /identity/v3/users => generated 285 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:02:01.249069 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18080924-b840-449d-a45d-9e6816a29666 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:01.258998 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18080924-b840-449d-a45d-9e6816a29666 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:01.263517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae9d536-4458-4ac8-8582-5da306f9913d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:01.264107 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9d536-4458-4ac8-8582-5da306f9913d admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:01.264382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9d536-4458-4ac8-8582-5da306f9913d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:01.264651 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9d536-4458-4ac8-8582-5da306f9913d admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:01.265283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae9d536-4458-4ac8-8582-5da306f9913d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:01.266907 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae9d536-4458-4ac8-8582-5da306f9913d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:01.277875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1396/2779] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:02:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:02:01.282161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca0adbb-2267-447e-b18a-0a2926b1f5a8 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:01.293819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca0adbb-2267-447e-b18a-0a2926b1f5a8 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:01.298548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb5e851-9f60-4681-bca5-09e6be63bf42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:01.299174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb5e851-9f60-4681-bca5-09e6be63bf42 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:01.299483 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb5e851-9f60-4681-bca5-09e6be63bf42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:01.299759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb5e851-9f60-4681-bca5-09e6be63bf42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:01.300433 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb5e851-9f60-4681-bca5-09e6be63bf42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:01.302147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb5e851-9f60-4681-bca5-09e6be63bf42 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:01.313623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1384/2780] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:02:01 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:02:01.317958 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae9d536-4458-4ac8-8582-5da306f9913d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:01.328302 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae9d536-4458-4ac8-8582-5da306f9913d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:01.334036 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03825c87-420b-4788-9050-df39e2559b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:01.334901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03825c87-420b-4788-9050-df39e2559b8e admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:01.335257 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03825c87-420b-4788-9050-df39e2559b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:01.335531 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03825c87-420b-4788-9050-df39e2559b8e admin admin] PATH_INFO: `/v3/projects/a8755c66b3e74383bf236abd8f2dd67f/users/69572df02ae743f09cdd59870622f149/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:01.351950 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03825c87-420b-4788-9050-df39e2559b8e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8755c66b3e74383bf236abd8f2dd67f, user_id=69572df02ae743f09cdd59870622f149, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:01.354642 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03825c87-420b-4788-9050-df39e2559b8e admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:01.367661 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1397/2781] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:02:01 2022] PUT /identity/v3/projects/a8755c66b3e74383bf236abd8f2dd67f/users/69572df02ae743f09cdd59870622f149/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:02:01.379713 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edf1390-5c47-4b99-a1cb-100df199374a None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:01.380026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edf1390-5c47-4b99-a1cb-100df199374a None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:01.380304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edf1390-5c47-4b99-a1cb-100df199374a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:01.408523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-8edf1390-5c47-4b99-a1cb-100df199374a None None] MFA Rules not processed for user `69572df02ae743f09cdd59870622f149`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:02:01.409732 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-8edf1390-5c47-4b99-a1cb-100df199374a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:01.476755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1385/2782] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:02:01 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:02:01.489542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03825c87-420b-4788-9050-df39e2559b8e admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:01.491395 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-03825c87-420b-4788-9050-df39e2559b8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:01.554016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03825c87-420b-4788-9050-df39e2559b8e admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:01.557725 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80ca374-4c23-47eb-aca8-0120d5f31b56 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc59f80f544cef8b3869e8d37bb341', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:01.558458 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80ca374-4c23-47eb-aca8-0120d5f31b56 service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:01.558837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80ca374-4c23-47eb-aca8-0120d5f31b56 service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:01.559151 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80ca374-4c23-47eb-aca8-0120d5f31b56 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:01.568367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80ca374-4c23-47eb-aca8-0120d5f31b56 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:01.569921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80ca374-4c23-47eb-aca8-0120d5f31b56 service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:01.579666 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1398/2783] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:02:01 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:02:11.473225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=46,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:11.575431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=3,keystone:DELETE=3,keystone:SELECT=59 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:27.597595 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e786c47a-791d-427f-9d1a-683e7302f0db None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:27.598126 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e786c47a-791d-427f-9d1a-683e7302f0db None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:27.598242 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e786c47a-791d-427f-9d1a-683e7302f0db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:27.617380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-e786c47a-791d-427f-9d1a-683e7302f0db None None] MFA Rules not processed for user `1750f2cc69c0499ebade616da183507b`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:02:27.618589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e786c47a-791d-427f-9d1a-683e7302f0db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:27.686234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1386/2784] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:02:27 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:02:37.664067 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=25 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:44.577140 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80ca374-4c23-47eb-aca8-0120d5f31b56 service glance] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:44.692207 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80ca374-4c23-47eb-aca8-0120d5f31b56 service glance] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:44.696862 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:44.697487 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:44.697762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:44.698085 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] PATH_INFO: `/v3/users/69572df02ae743f09cdd59870622f149` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:44.699218 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] RBAC: Authorizing `identity:delete_user(user_id=69572df02ae743f09cdd59870622f149)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:44.701154 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:44.749182 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '69572df02ae743f09cdd59870622f149'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:44.756053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '69572df02ae743f09cdd59870622f149'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:44.756705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '69572df02ae743f09cdd59870622f149'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:44.757298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '69572df02ae743f09cdd59870622f149'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:44.764389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '69572df02ae743f09cdd59870622f149'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:44.782272 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1399/2785] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:02:44 2022] DELETE /identity/v3/users/69572df02ae743f09cdd59870622f149 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:02:44.786841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394fb282-e5d4-4b12-b1e7-a923b8b47014 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:44.787319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394fb282-e5d4-4b12-b1e7-a923b8b47014 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:44.787425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394fb282-e5d4-4b12-b1e7-a923b8b47014 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:44.837144 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-394fb282-e5d4-4b12-b1e7-a923b8b47014 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:02:44.838375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-394fb282-e5d4-4b12-b1e7-a923b8b47014 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:44.896792 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1387/2786] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:02:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:02:44.906383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:44.907789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:44.949074 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:44.983805 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7675d16-5e15-4ebf-b584-a92c332a6fdb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:44.987458 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24a0bf4-66b6-4fc5-81e1-0c62d078b1af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:44.988054 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a0bf4-66b6-4fc5-81e1-0c62d078b1af service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:44.988579 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a0bf4-66b6-4fc5-81e1-0c62d078b1af service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:44.988864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24a0bf4-66b6-4fc5-81e1-0c62d078b1af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:44.997737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24a0bf4-66b6-4fc5-81e1-0c62d078b1af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:44.999205 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24a0bf4-66b6-4fc5-81e1-0c62d078b1af service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:45.008484 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1400/2787] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:02:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:02:45.276724 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394fb282-e5d4-4b12-b1e7-a923b8b47014 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:45.278281 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-394fb282-e5d4-4b12-b1e7-a923b8b47014 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:45.292825 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394fb282-e5d4-4b12-b1e7-a923b8b47014 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:45.298729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:45.299481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:45.299777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:45.300026 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] PATH_INFO: `/v3/projects/a8755c66b3e74383bf236abd8f2dd67f` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:45.301928 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8755c66b3e74383bf236abd8f2dd67f)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:45.304411 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:45.367594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8755c66b3e74383bf236abd8f2dd67f', 'request_id': 'req-c0b96cd2-5132-4225-8948-eb3af4eccae6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:45.376099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8755c66b3e74383bf236abd8f2dd67f', 'request_id': 'req-c0b96cd2-5132-4225-8948-eb3af4eccae6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:45.386094 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8755c66b3e74383bf236abd8f2dd67f', 'request_id': 'req-c0b96cd2-5132-4225-8948-eb3af4eccae6'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:45.397228 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] The token cache is being invalidate because project a8755c66b3e74383bf236abd8f2dd67f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:02:45.397577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:45.398265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-c0b96cd2-5132-4225-8948-eb3af4eccae6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:02:45.399828 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1388/2788] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:02:45 2022] DELETE /identity/v3/projects/a8755c66b3e74383bf236abd8f2dd67f => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 01:02:45.405722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2b43a4-9595-4fdc-94b3-0d853f10192b None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:45.406005 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2b43a4-9595-4fdc-94b3-0d853f10192b None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:45.406308 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2b43a4-9595-4fdc-94b3-0d853f10192b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:45.453234 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4f2b43a4-9595-4fdc-94b3-0d853f10192b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:02:45.454474 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4f2b43a4-9595-4fdc-94b3-0d853f10192b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:45.511294 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1401/2789] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:02:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:02:45.519368 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a977cbc8-13e8-4d62-b7ba-df3298ae34e1 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:45.519651 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a977cbc8-13e8-4d62-b7ba-df3298ae34e1 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:45.519946 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a977cbc8-13e8-4d62-b7ba-df3298ae34e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:45.539707 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a977cbc8-13e8-4d62-b7ba-df3298ae34e1 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:02:45.540891 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a977cbc8-13e8-4d62-b7ba-df3298ae34e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:45.549594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1389/2790] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:02:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:02:45.553699 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2b43a4-9595-4fdc-94b3-0d853f10192b None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:45.563061 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2b43a4-9595-4fdc-94b3-0d853f10192b None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:45.567589 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec16173-1f90-4266-a02a-431d79bbb307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:45.568169 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec16173-1f90-4266-a02a-431d79bbb307 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:45.568501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec16173-1f90-4266-a02a-431d79bbb307 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:45.568776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec16173-1f90-4266-a02a-431d79bbb307 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:45.569400 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec16173-1f90-4266-a02a-431d79bbb307 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:02:45.569693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec16173-1f90-4266-a02a-431d79bbb307 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:45.571402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec16173-1f90-4266-a02a-431d79bbb307 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:45.583407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1402/2791] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:02:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:02:45.587503 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a977cbc8-13e8-4d62-b7ba-df3298ae34e1 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:45.596966 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a977cbc8-13e8-4d62-b7ba-df3298ae34e1 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:45.601481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c0b6b0-1818-4e3f-a753-f877c097b0de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:45.602145 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c0b6b0-1818-4e3f-a753-f877c097b0de admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:45.602359 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c0b6b0-1818-4e3f-a753-f877c097b0de admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:45.602626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c0b6b0-1818-4e3f-a753-f877c097b0de admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:45.603252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c0b6b0-1818-4e3f-a753-f877c097b0de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:45.605104 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c0b6b0-1818-4e3f-a753-f877c097b0de admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:45.632628 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1390/2792] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:02:45 2022] POST /identity/v3/projects => generated 377 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:02:45.637301 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec16173-1f90-4266-a02a-431d79bbb307 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:45.647077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec16173-1f90-4266-a02a-431d79bbb307 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:45.702621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a74389-23c6-4349-b806-f27dcc360236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:45.703217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a74389-23c6-4349-b806-f27dcc360236 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:45.703496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a74389-23c6-4349-b806-f27dcc360236 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:45.703771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a74389-23c6-4349-b806-f27dcc360236 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:45.704389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a74389-23c6-4349-b806-f27dcc360236 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:45.706376 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a74389-23c6-4349-b806-f27dcc360236 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:45.727298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1403/2793] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:02:45 2022] POST /identity/v3/users => generated 289 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:02:45.731293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c0b6b0-1818-4e3f-a753-f877c097b0de admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:45.741494 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c0b6b0-1818-4e3f-a753-f877c097b0de admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:45.746611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5eda1b8-f40c-4760-a420-cf3c802139d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:45.747200 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5eda1b8-f40c-4760-a420-cf3c802139d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:45.747481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5eda1b8-f40c-4760-a420-cf3c802139d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:45.747755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5eda1b8-f40c-4760-a420-cf3c802139d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:45.748404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5eda1b8-f40c-4760-a420-cf3c802139d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:45.749961 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5eda1b8-f40c-4760-a420-cf3c802139d5 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:45.759656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1391/2794] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:02:45 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:02:45.763935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a74389-23c6-4349-b806-f27dcc360236 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:45.774414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a74389-23c6-4349-b806-f27dcc360236 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:45.779715 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87810a97-6b13-419b-ac3c-e50c42ffc9c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:45.780300 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87810a97-6b13-419b-ac3c-e50c42ffc9c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:45.780572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87810a97-6b13-419b-ac3c-e50c42ffc9c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:45.780837 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87810a97-6b13-419b-ac3c-e50c42ffc9c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:45.781456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87810a97-6b13-419b-ac3c-e50c42ffc9c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:45.783060 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87810a97-6b13-419b-ac3c-e50c42ffc9c2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:45.793409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1404/2795] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:02:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:02:45.797346 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5eda1b8-f40c-4760-a420-cf3c802139d5 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:45.807096 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5eda1b8-f40c-4760-a420-cf3c802139d5 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:45.812371 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4120823f-d6d7-45df-8cce-e83837349837 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:45.813073 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4120823f-d6d7-45df-8cce-e83837349837 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:45.813377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4120823f-d6d7-45df-8cce-e83837349837 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:45.813652 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4120823f-d6d7-45df-8cce-e83837349837 admin admin] PATH_INFO: `/v3/projects/017c596a2d0f430d987352a616666610/users/a94a5f91289c403cb49f428e53431591/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:45.828547 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4120823f-d6d7-45df-8cce-e83837349837 admin admin] RBAC: Authorizing `identity:create_grant(project_id=017c596a2d0f430d987352a616666610, user_id=a94a5f91289c403cb49f428e53431591, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:45.831197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4120823f-d6d7-45df-8cce-e83837349837 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:45.833299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:45.845742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1392/2796] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:02:45 2022] PUT /identity/v3/projects/017c596a2d0f430d987352a616666610/users/a94a5f91289c403cb49f428e53431591/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:02:45.850719 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a15320f-aa6c-470f-9624-2cec624f996d None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:45.850997 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a15320f-aa6c-470f-9624-2cec624f996d None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:45.851267 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a15320f-aa6c-470f-9624-2cec624f996d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:45.864720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:45.871195 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-9a15320f-aa6c-470f-9624-2cec624f996d None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:02:45.872389 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-9a15320f-aa6c-470f-9624-2cec624f996d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:45.928672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1405/2797] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:02:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:02:45.938847 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4120823f-d6d7-45df-8cce-e83837349837 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:45.940536 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-4120823f-d6d7-45df-8cce-e83837349837 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:45.997013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4120823f-d6d7-45df-8cce-e83837349837 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:46.000640 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8d85e-b62d-4e20-95c4-949376c3b9f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:46.001225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8d85e-b62d-4e20-95c4-949376c3b9f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:46.001499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8d85e-b62d-4e20-95c4-949376c3b9f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:46.001766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d8d85e-b62d-4e20-95c4-949376c3b9f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:46.011133 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d8d85e-b62d-4e20-95c4-949376c3b9f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:46.012573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d8d85e-b62d-4e20-95c4-949376c3b9f0 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:46.021974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1393/2798] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:02:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:02:49.956782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae51352c-b1e4-489d-b11b-a6f099979516 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:49.957359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae51352c-b1e4-489d-b11b-a6f099979516 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:49.957359 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae51352c-b1e4-489d-b11b-a6f099979516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:49.986557 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-ae51352c-b1e4-489d-b11b-a6f099979516 None None] MFA Rules not processed for user `a94a5f91289c403cb49f428e53431591`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:02:49.987692 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ae51352c-b1e4-489d-b11b-a6f099979516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:50.059115 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1406/2799] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:02:49 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:02:50.063784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8d85e-b62d-4e20-95c4-949376c3b9f0 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:50.074286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d8d85e-b62d-4e20-95c4-949376c3b9f0 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:50.079244 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f344b8d-38cd-4a1e-89f0-18e2a3dfb381 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:50.079841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f344b8d-38cd-4a1e-89f0-18e2a3dfb381 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:50.080114 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f344b8d-38cd-4a1e-89f0-18e2a3dfb381 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:50.080383 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f344b8d-38cd-4a1e-89f0-18e2a3dfb381 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:50.081015 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f344b8d-38cd-4a1e-89f0-18e2a3dfb381 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:50.082927 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f344b8d-38cd-4a1e-89f0-18e2a3dfb381 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:50.111500 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1394/2800] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:02:50 2022] POST /identity/v3/projects => generated 379 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:02:50.115688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae51352c-b1e4-489d-b11b-a6f099979516 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:50.127762 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae51352c-b1e4-489d-b11b-a6f099979516 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:50.220784 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328cddbf-9d39-446f-be34-9883b4a2623d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:50.221393 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328cddbf-9d39-446f-be34-9883b4a2623d admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:50.221672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328cddbf-9d39-446f-be34-9883b4a2623d admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:50.221945 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328cddbf-9d39-446f-be34-9883b4a2623d admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:50.222626 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328cddbf-9d39-446f-be34-9883b4a2623d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:50.224541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328cddbf-9d39-446f-be34-9883b4a2623d admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:50.256984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1407/2801] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:02:50 2022] POST /identity/v3/users => generated 296 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:02:50.258479 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f344b8d-38cd-4a1e-89f0-18e2a3dfb381 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:50.269007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f344b8d-38cd-4a1e-89f0-18e2a3dfb381 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:50.273916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87ceffd-9a9f-4877-a59f-c7bbcc59dac1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:50.274586 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87ceffd-9a9f-4877-a59f-c7bbcc59dac1 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:50.274857 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87ceffd-9a9f-4877-a59f-c7bbcc59dac1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:50.275121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87ceffd-9a9f-4877-a59f-c7bbcc59dac1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:50.276377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87ceffd-9a9f-4877-a59f-c7bbcc59dac1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:50.277996 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87ceffd-9a9f-4877-a59f-c7bbcc59dac1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:50.289568 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1395/2802] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:02:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:02:50.294098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328cddbf-9d39-446f-be34-9883b4a2623d admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:50.308925 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328cddbf-9d39-446f-be34-9883b4a2623d admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:50.314199 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faaa767-be13-4b81-9724-b45174a1a2fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:50.314876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faaa767-be13-4b81-9724-b45174a1a2fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:50.315158 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faaa767-be13-4b81-9724-b45174a1a2fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:50.315431 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faaa767-be13-4b81-9724-b45174a1a2fa admin admin] PATH_INFO: `/v3/projects/4512303a95db4c8cbf896ca76724a559/users/73e4f22128c24c27ad5d3a0a48435de7/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:50.331417 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faaa767-be13-4b81-9724-b45174a1a2fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=4512303a95db4c8cbf896ca76724a559, user_id=73e4f22128c24c27ad5d3a0a48435de7, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:50.334192 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faaa767-be13-4b81-9724-b45174a1a2fa admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:50.348892 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1408/2803] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:02:50 2022] PUT /identity/v3/projects/4512303a95db4c8cbf896ca76724a559/users/73e4f22128c24c27ad5d3a0a48435de7/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 01:02:50.353257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87ceffd-9a9f-4877-a59f-c7bbcc59dac1 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:50.364573 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87ceffd-9a9f-4877-a59f-c7bbcc59dac1 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:50.423516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c301976-3ecb-4905-b0b7-805defca9ed2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:50.424129 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c301976-3ecb-4905-b0b7-805defca9ed2 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:50.424404 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c301976-3ecb-4905-b0b7-805defca9ed2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:50.424672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c301976-3ecb-4905-b0b7-805defca9ed2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:50.425302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c301976-3ecb-4905-b0b7-805defca9ed2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:50.426939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c301976-3ecb-4905-b0b7-805defca9ed2 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:50.437287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1396/2804] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:02:50 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:02:50.441388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faaa767-be13-4b81-9724-b45174a1a2fa admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:50.447022 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:50.453761 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faaa767-be13-4b81-9724-b45174a1a2fa admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:50.458658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f52b658-1c9b-4473-b98f-03a73a5b8907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:50.459316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f52b658-1c9b-4473-b98f-03a73a5b8907 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:50.459586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f52b658-1c9b-4473-b98f-03a73a5b8907 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:50.459853 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f52b658-1c9b-4473-b98f-03a73a5b8907 admin admin] PATH_INFO: `/v3/projects/4512303a95db4c8cbf896ca76724a559/users/73e4f22128c24c27ad5d3a0a48435de7/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:50.461751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f52b658-1c9b-4473-b98f-03a73a5b8907 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4512303a95db4c8cbf896ca76724a559, user_id=73e4f22128c24c27ad5d3a0a48435de7, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:50.465014 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f52b658-1c9b-4473-b98f-03a73a5b8907 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:50.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1409/2805] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:02:50 2022] PUT /identity/v3/projects/4512303a95db4c8cbf896ca76724a559/users/73e4f22128c24c27ad5d3a0a48435de7/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:02:55.310809 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbbee1f-4ebd-438d-b84f-477423b18a15 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:55.311304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbbee1f-4ebd-438d-b84f-477423b18a15 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:55.311556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbbee1f-4ebd-438d-b84f-477423b18a15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:55.343362 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ecbbee1f-4ebd-438d-b84f-477423b18a15 None None] MFA Rules not processed for user `73e4f22128c24c27ad5d3a0a48435de7`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:02:55.346557 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ecbbee1f-4ebd-438d-b84f-477423b18a15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:55.422099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:02:55.459420 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1397/2806] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:02:55 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:02:55.473901 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f52b658-1c9b-4473-b98f-03a73a5b8907 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:55.475811 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4f52b658-1c9b-4473-b98f-03a73a5b8907 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:55.588357 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f52b658-1c9b-4473-b98f-03a73a5b8907 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:55.594316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db3b102-fc01-44e9-9931-4b4e3ef85a44 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:55.594943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3b102-fc01-44e9-9931-4b4e3ef85a44 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:55.595222 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3b102-fc01-44e9-9931-4b4e3ef85a44 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:55.595496 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db3b102-fc01-44e9-9931-4b4e3ef85a44 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:55.656939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3b102-fc01-44e9-9931-4b4e3ef85a44 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:55.658610 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db3b102-fc01-44e9-9931-4b4e3ef85a44 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:55.667954 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1410/2807] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:02:55 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:02:57.279113 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbbee1f-4ebd-438d-b84f-477423b18a15 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:02:57.281854 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ecbbee1f-4ebd-438d-b84f-477423b18a15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:57.402198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbbee1f-4ebd-438d-b84f-477423b18a15 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:02:57.407646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177d8319-d8e5-49c2-ae0b-b648906cb807 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:02:57.408255 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177d8319-d8e5-49c2-ae0b-b648906cb807 service placement] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:02:57.408529 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177d8319-d8e5-49c2-ae0b-b648906cb807 service placement] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:02:57.408795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177d8319-d8e5-49c2-ae0b-b648906cb807 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:02:57.410597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-177d8319-d8e5-49c2-ae0b-b648906cb807 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:02:57.425306 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177d8319-d8e5-49c2-ae0b-b648906cb807 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:02:57.427198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177d8319-d8e5-49c2-ae0b-b648906cb807 service placement] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:02:57.437887 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1398/2808] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:02:57 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:03:05.663594 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=72,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:03:07.433686 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=48 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:03:08.051564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b11ced-f40c-4347-b078-bb29a53a118c None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:03:08.052089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b11ced-f40c-4347-b078-bb29a53a118c None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:03:08.052089 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b11ced-f40c-4347-b078-bb29a53a118c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:03:08.071905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-63b11ced-f40c-4347-b078-bb29a53a118c None None] MFA Rules not processed for user `a84c6036067e4ed2817f56d95eaf1715`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:03:08.073103 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-63b11ced-f40c-4347-b078-bb29a53a118c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:03:08.127750 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1411/2809] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:03:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:03:13.359561 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cba91f-a8fd-4f0a-a46c-080763cdffb4 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:03:13.360076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cba91f-a8fd-4f0a-a46c-080763cdffb4 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:03:13.360205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cba91f-a8fd-4f0a-a46c-080763cdffb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:03:13.380273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-d2cba91f-a8fd-4f0a-a46c-080763cdffb4 None None] MFA Rules not processed for user `a94a5f91289c403cb49f428e53431591`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:03:13.381481 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-d2cba91f-a8fd-4f0a-a46c-080763cdffb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:03:13.388812 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1399/2810] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:03:13 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:03:18.121015 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=25 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:03:23.375159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:03:35.629438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b11ced-f40c-4347-b078-bb29a53a118c None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:03:35.631216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-63b11ced-f40c-4347-b078-bb29a53a118c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:03:35.719122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b11ced-f40c-4347-b078-bb29a53a118c None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:03:35.722549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10ff7da-7913-4e30-bea0-f17b7151a75e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:03:35.723220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10ff7da-7913-4e30-bea0-f17b7151a75e service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:03:35.723568 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10ff7da-7913-4e30-bea0-f17b7151a75e service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:03:35.723839 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10ff7da-7913-4e30-bea0-f17b7151a75e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:03:35.734665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10ff7da-7913-4e30-bea0-f17b7151a75e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:03:35.736233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10ff7da-7913-4e30-bea0-f17b7151a75e service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:03:35.746831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1412/2811] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:03:35 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:03:45.605348 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551cc20-f1a1-46ce-abab-3151640c2621 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:03:45.606020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551cc20-f1a1-46ce-abab-3151640c2621 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:03:45.606020 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1551cc20-f1a1-46ce-abab-3151640c2621 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:03:45.626785 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-1551cc20-f1a1-46ce-abab-3151640c2621 None None] MFA Rules not processed for user `1750f2cc69c0499ebade616da183507b`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:03:45.627976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1551cc20-f1a1-46ce-abab-3151640c2621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:03:45.696462 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1400/2812] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:03:45 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:03:45.741780 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=35 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:03:55.689777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=25 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:04:07.204964 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10ff7da-7913-4e30-bea0-f17b7151a75e service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:04:07.216703 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10ff7da-7913-4e30-bea0-f17b7151a75e service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:04:07.221667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b05225e-316b-4e1f-a395-782e72c1b746 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:04:07.222280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b05225e-316b-4e1f-a395-782e72c1b746 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:04:07.222556 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b05225e-316b-4e1f-a395-782e72c1b746 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:04:07.222832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b05225e-316b-4e1f-a395-782e72c1b746 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:04:07.224540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-7b05225e-316b-4e1f-a395-782e72c1b746 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:04:07.235836 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b05225e-316b-4e1f-a395-782e72c1b746 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:04:07.237663 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b05225e-316b-4e1f-a395-782e72c1b746 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:04:07.250203 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1413/2813] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:04:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 02 01:04:07.261276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1551cc20-f1a1-46ce-abab-3151640c2621 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:04:07.262805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1551cc20-f1a1-46ce-abab-3151640c2621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:04:07.330013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1551cc20-f1a1-46ce-abab-3151640c2621 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:04:07.334521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:04:07.335138 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:04:07.335443 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:04:07.335712 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] PATH_INFO: `/v3/users/a84c6036067e4ed2817f56d95eaf1715` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:04:07.336804 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a84c6036067e4ed2817f56d95eaf1715)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:04:07.339473 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:04:07.385203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a84c6036067e4ed2817f56d95eaf1715'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:04:07.385848 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a84c6036067e4ed2817f56d95eaf1715'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:04:07.392626 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a84c6036067e4ed2817f56d95eaf1715'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:04:07.393246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a84c6036067e4ed2817f56d95eaf1715'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:04:07.401684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a84c6036067e4ed2817f56d95eaf1715'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:04:07.420038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1401/2814] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:04:07 2022] DELETE /identity/v3/users/a84c6036067e4ed2817f56d95eaf1715 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:04:07.586407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b05225e-316b-4e1f-a395-782e72c1b746 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:04:07.600386 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b05225e-316b-4e1f-a395-782e72c1b746 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:04:07.656643 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:04:07.657261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:04:07.657533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:04:07.657800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] PATH_INFO: `/v3/projects/31b72d052142491dbfc1936f4f006715` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:04:07.658918 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] RBAC: Authorizing `identity:delete_project(project_id=31b72d052142491dbfc1936f4f006715)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:04:07.660803 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:04:07.704611 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '31b72d052142491dbfc1936f4f006715', 'request_id': 'req-79f2c130-7220-4566-a414-5bf61002f333'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:04:07.711458 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '31b72d052142491dbfc1936f4f006715', 'request_id': 'req-79f2c130-7220-4566-a414-5bf61002f333'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:04:07.717906 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '31b72d052142491dbfc1936f4f006715', 'request_id': 'req-79f2c130-7220-4566-a414-5bf61002f333'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:04:07.728477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] The token cache is being invalidate because project 31b72d052142491dbfc1936f4f006715 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:04:07.728756 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:04:07.729370 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:04:07.730835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1414/2815] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:04:07 2022] DELETE /identity/v3/projects/31b72d052142491dbfc1936f4f006715 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:04:17.416530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=53,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:04:17.724965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=57,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:04:41.018396 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:04:41.019777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:04:41.098288 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386c044b-a09c-400f-9b69-a73f174f673e admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:04:41.145923 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186e01cb-eb03-4b46-9b21-55f29eee4b7f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:04:41.146484 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186e01cb-eb03-4b46-9b21-55f29eee4b7f service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:04:41.146764 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186e01cb-eb03-4b46-9b21-55f29eee4b7f service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:04:41.147038 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186e01cb-eb03-4b46-9b21-55f29eee4b7f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:04:41.148770 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-186e01cb-eb03-4b46-9b21-55f29eee4b7f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:04:41.217010 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186e01cb-eb03-4b46-9b21-55f29eee4b7f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:04:41.218478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186e01cb-eb03-4b46-9b21-55f29eee4b7f service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:04:41.229185 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1402/2816] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:04:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:04:49.044511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:04:49.045904 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:04:49.058146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f2c130-7220-4566-a414-5bf61002f333 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:04:49.061627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40293081-b3f2-4083-bb83-a24c661fc956 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:04:49.062248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40293081-b3f2-4083-bb83-a24c661fc956 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:04:49.062522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40293081-b3f2-4083-bb83-a24c661fc956 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:04:49.062791 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40293081-b3f2-4083-bb83-a24c661fc956 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:04:49.064432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-40293081-b3f2-4083-bb83-a24c661fc956 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:04:49.077665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40293081-b3f2-4083-bb83-a24c661fc956 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:04:49.079105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40293081-b3f2-4083-bb83-a24c661fc956 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:04:49.090394 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1415/2817] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:04:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:04:51.224436 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=83 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:04:57.204373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f360b2-8336-4ed2-aa55-7db6f6b531f4 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:04:57.204945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f360b2-8336-4ed2-aa55-7db6f6b531f4 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:04:57.204945 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f360b2-8336-4ed2-aa55-7db6f6b531f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:04:57.223579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-c8f360b2-8336-4ed2-aa55-7db6f6b531f4 None None] MFA Rules not processed for user `1750f2cc69c0499ebade616da183507b`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:04:57.225243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c8f360b2-8336-4ed2-aa55-7db6f6b531f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:04:57.278486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1403/2818] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:04:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:04:59.085003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:05:00.677972 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40293081-b3f2-4083-bb83-a24c661fc956 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:05:00.679435 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-40293081-b3f2-4083-bb83-a24c661fc956 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:05:00.734282 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40293081-b3f2-4083-bb83-a24c661fc956 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:05:00.738176 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d53bc10-4d2f-4bee-a404-75ad91ff614a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:05:00.738768 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d53bc10-4d2f-4bee-a404-75ad91ff614a service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:05:00.739046 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d53bc10-4d2f-4bee-a404-75ad91ff614a service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:05:00.739310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d53bc10-4d2f-4bee-a404-75ad91ff614a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:05:00.741000 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5d53bc10-4d2f-4bee-a404-75ad91ff614a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:05:00.750989 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d53bc10-4d2f-4bee-a404-75ad91ff614a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:05:00.752468 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d53bc10-4d2f-4bee-a404-75ad91ff614a service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:05:00.761048 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1416/2819] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:05:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:05:07.271841 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=25 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:05:10.756824 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=26 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:05:38.541693 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f360b2-8336-4ed2-aa55-7db6f6b531f4 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:05:38.543811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c8f360b2-8336-4ed2-aa55-7db6f6b531f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:05:38.560906 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f360b2-8336-4ed2-aa55-7db6f6b531f4 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:05:38.570281 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e6b987-0f5f-4938-8017-0967c096e8c8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:05:38.570949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e6b987-0f5f-4938-8017-0967c096e8c8 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:05:38.571672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e6b987-0f5f-4938-8017-0967c096e8c8 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:05:38.572057 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e6b987-0f5f-4938-8017-0967c096e8c8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:05:38.574577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a5e6b987-0f5f-4938-8017-0967c096e8c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:05:38.587938 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e6b987-0f5f-4938-8017-0967c096e8c8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:05:38.589939 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e6b987-0f5f-4938-8017-0967c096e8c8 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:05:38.601630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1404/2820] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:05:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:05:40.507010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aac023-cf4e-46fd-93de-25539571a976 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:05:40.507488 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aac023-cf4e-46fd-93de-25539571a976 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:05:40.507596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aac023-cf4e-46fd-93de-25539571a976 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:05:40.527600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a9aac023-cf4e-46fd-93de-25539571a976 None None] MFA Rules not processed for user `a94a5f91289c403cb49f428e53431591`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:05:40.528789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a9aac023-cf4e-46fd-93de-25539571a976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:05:40.581299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1417/2821] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:05:40 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:05:41.333524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e6b987-0f5f-4938-8017-0967c096e8c8 service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:05:41.335016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a5e6b987-0f5f-4938-8017-0967c096e8c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:05:41.403599 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e6b987-0f5f-4938-8017-0967c096e8c8 service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:05:41.408520 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad756fe-91c0-4c86-904e-1837f48101e4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:05:41.409121 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad756fe-91c0-4c86-904e-1837f48101e4 service placement] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:05:41.409425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad756fe-91c0-4c86-904e-1837f48101e4 service placement] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:05:41.409696 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad756fe-91c0-4c86-904e-1837f48101e4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:05:41.411422 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5ad756fe-91c0-4c86-904e-1837f48101e4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:05:41.422903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad756fe-91c0-4c86-904e-1837f48101e4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:05:41.424489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad756fe-91c0-4c86-904e-1837f48101e4 service placement] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:05:41.434644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1405/2822] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:05:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:05:50.574737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=25 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:05:51.429907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=38 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:05:55.864309 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aac023-cf4e-46fd-93de-25539571a976 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:05:55.881741 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aac023-cf4e-46fd-93de-25539571a976 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:05:55.887415 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d80ceeb-60f8-405e-ae9e-dbcaa781a3eb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:05:55.888633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d80ceeb-60f8-405e-ae9e-dbcaa781a3eb service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:05:55.889109 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d80ceeb-60f8-405e-ae9e-dbcaa781a3eb service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:05:55.889447 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d80ceeb-60f8-405e-ae9e-dbcaa781a3eb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:05:55.891289 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2d80ceeb-60f8-405e-ae9e-dbcaa781a3eb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:05:55.905513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d80ceeb-60f8-405e-ae9e-dbcaa781a3eb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:05:55.907872 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d80ceeb-60f8-405e-ae9e-dbcaa781a3eb service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:05:55.920184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1418/2823] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:05:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:06:05.915931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:10.057110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad756fe-91c0-4c86-904e-1837f48101e4 service placement] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:10.058691 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5ad756fe-91c0-4c86-904e-1837f48101e4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:10.125558 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad756fe-91c0-4c86-904e-1837f48101e4 service placement] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:10.130196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:10.130816 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.131090 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.131365 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] PATH_INFO: `/v3/users/1750f2cc69c0499ebade616da183507b` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:10.132456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] RBAC: Authorizing `identity:delete_user(user_id=1750f2cc69c0499ebade616da183507b)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:10.134439 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:10.180941 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1750f2cc69c0499ebade616da183507b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:10.181523 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1750f2cc69c0499ebade616da183507b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:10.188286 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1750f2cc69c0499ebade616da183507b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:10.188908 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1750f2cc69c0499ebade616da183507b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:10.195671 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7b4aa8db-b8a4-45a2-aecd-6d13fc885f0c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1750f2cc69c0499ebade616da183507b'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:10.212955 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1406/2824] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:06:10 2022] DELETE /identity/v3/users/1750f2cc69c0499ebade616da183507b => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:10.379471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d80ceeb-60f8-405e-ae9e-dbcaa781a3eb service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:10.391713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d80ceeb-60f8-405e-ae9e-dbcaa781a3eb service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:10.448025 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:10.448575 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.448851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.449153 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] PATH_INFO: `/v3/projects/29870f31b41846cc923a28ebfaf368db` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:10.450465 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] RBAC: Authorizing `identity:delete_project(project_id=29870f31b41846cc923a28ebfaf368db)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:10.452878 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:10.498162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29870f31b41846cc923a28ebfaf368db', 'request_id': 'req-5d252545-7342-456d-80c7-147f1663cc63'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:10.505097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29870f31b41846cc923a28ebfaf368db', 'request_id': 'req-5d252545-7342-456d-80c7-147f1663cc63'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:10.511370 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29870f31b41846cc923a28ebfaf368db', 'request_id': 'req-5d252545-7342-456d-80c7-147f1663cc63'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:10.522331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] The token cache is being invalidate because project 29870f31b41846cc923a28ebfaf368db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:06:10.522615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:10.523236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5d252545-7342-456d-80c7-147f1663cc63 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:10.524712 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1419/2825] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:06:10 2022] DELETE /identity/v3/projects/29870f31b41846cc923a28ebfaf368db => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:10.529722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01af301-5bc7-4d06-a2ee-a421c456def5 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.530115 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01af301-5bc7-4d06-a2ee-a421c456def5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.530495 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01af301-5bc7-4d06-a2ee-a421c456def5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:10.550044 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-a01af301-5bc7-4d06-a2ee-a421c456def5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:10.551312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a01af301-5bc7-4d06-a2ee-a421c456def5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:10.610559 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1407/2826] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:10.618620 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e6b80d-4c4c-44b5-a73e-89c1c9f3b721 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.618903 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e6b80d-4c4c-44b5-a73e-89c1c9f3b721 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.619173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e6b80d-4c4c-44b5-a73e-89c1c9f3b721 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:10.638652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-54e6b80d-4c4c-44b5-a73e-89c1c9f3b721 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:10.639835 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-54e6b80d-4c4c-44b5-a73e-89c1c9f3b721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:10.647528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1420/2827] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:10.651489 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01af301-5bc7-4d06-a2ee-a421c456def5 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:10.661021 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01af301-5bc7-4d06-a2ee-a421c456def5 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:10.665521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53aa0a4e-65f3-4255-8021-6b5f14ae4831 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:10.666216 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53aa0a4e-65f3-4255-8021-6b5f14ae4831 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.666496 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53aa0a4e-65f3-4255-8021-6b5f14ae4831 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.666766 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53aa0a4e-65f3-4255-8021-6b5f14ae4831 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:10.667399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53aa0a4e-65f3-4255-8021-6b5f14ae4831 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:06:10.667699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53aa0a4e-65f3-4255-8021-6b5f14ae4831 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:10.669803 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53aa0a4e-65f3-4255-8021-6b5f14ae4831 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:10.681398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1408/2828] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:06:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:06:10.686142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e6b80d-4c4c-44b5-a73e-89c1c9f3b721 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:10.695120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e6b80d-4c4c-44b5-a73e-89c1c9f3b721 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:10.699759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbd57f7-15ee-4325-bc3f-74581591eb74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:10.700283 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbd57f7-15ee-4325-bc3f-74581591eb74 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.700558 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbd57f7-15ee-4325-bc3f-74581591eb74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.700827 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbd57f7-15ee-4325-bc3f-74581591eb74 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:10.701443 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbd57f7-15ee-4325-bc3f-74581591eb74 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:10.703355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbd57f7-15ee-4325-bc3f-74581591eb74 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:10.750499 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1421/2829] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:06:10 2022] POST /identity/v3/projects => generated 357 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:06:10.754974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53aa0a4e-65f3-4255-8021-6b5f14ae4831 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:10.764997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53aa0a4e-65f3-4255-8021-6b5f14ae4831 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:10.768549 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:10.825564 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66c1654-3b5b-4230-bec6-6a7a59781591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:10.826205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66c1654-3b5b-4230-bec6-6a7a59781591 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.826480 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66c1654-3b5b-4230-bec6-6a7a59781591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.826754 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66c1654-3b5b-4230-bec6-6a7a59781591 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:10.827386 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66c1654-3b5b-4230-bec6-6a7a59781591 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:10.829287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66c1654-3b5b-4230-bec6-6a7a59781591 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:10.851678 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1409/2830] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:06:10 2022] POST /identity/v3/users => generated 279 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:06:10.855887 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbd57f7-15ee-4325-bc3f-74581591eb74 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:10.866633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbd57f7-15ee-4325-bc3f-74581591eb74 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:10.871098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de203ef-5e6f-4105-8317-fcc91c7592c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:10.871685 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de203ef-5e6f-4105-8317-fcc91c7592c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.871955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de203ef-5e6f-4105-8317-fcc91c7592c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.872798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de203ef-5e6f-4105-8317-fcc91c7592c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:10.873457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de203ef-5e6f-4105-8317-fcc91c7592c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:10.875164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de203ef-5e6f-4105-8317-fcc91c7592c6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:10.884794 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1422/2831] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:10 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:10.888709 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66c1654-3b5b-4230-bec6-6a7a59781591 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:10.899723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66c1654-3b5b-4230-bec6-6a7a59781591 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:10.904342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49fb9521-6d85-4080-a275-1bf0f0db1d50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:10.905332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fb9521-6d85-4080-a275-1bf0f0db1d50 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.905332 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fb9521-6d85-4080-a275-1bf0f0db1d50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.905554 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49fb9521-6d85-4080-a275-1bf0f0db1d50 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:10.906097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fb9521-6d85-4080-a275-1bf0f0db1d50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:10.907739 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49fb9521-6d85-4080-a275-1bf0f0db1d50 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:10.918735 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1410/2832] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:10.924102 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de203ef-5e6f-4105-8317-fcc91c7592c6 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:10.933875 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de203ef-5e6f-4105-8317-fcc91c7592c6 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:10.939239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b938c95-c5b0-40b3-96fc-90570be05228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:10.940118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b938c95-c5b0-40b3-96fc-90570be05228 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.940118 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b938c95-c5b0-40b3-96fc-90570be05228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.940382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b938c95-c5b0-40b3-96fc-90570be05228 admin admin] PATH_INFO: `/v3/projects/c9b47dd257be4ee4b11107c5ba9b734b/users/78004c7d5d424ed597c9d79e2a7b1f5e/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:10.956409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b938c95-c5b0-40b3-96fc-90570be05228 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9b47dd257be4ee4b11107c5ba9b734b, user_id=78004c7d5d424ed597c9d79e2a7b1f5e, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:10.959119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b938c95-c5b0-40b3-96fc-90570be05228 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:10.973295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1423/2833] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:06:10 2022] PUT /identity/v3/projects/c9b47dd257be4ee4b11107c5ba9b734b/users/78004c7d5d424ed597c9d79e2a7b1f5e/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:10.984090 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed365582-94fd-4d1c-91e9-beae5db25f1f None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:10.984467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed365582-94fd-4d1c-91e9-beae5db25f1f None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:10.984808 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed365582-94fd-4d1c-91e9-beae5db25f1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:11.013895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-ed365582-94fd-4d1c-91e9-beae5db25f1f None None] MFA Rules not processed for user `78004c7d5d424ed597c9d79e2a7b1f5e`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:11.015150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-ed365582-94fd-4d1c-91e9-beae5db25f1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:11.083589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1411/2834] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:10 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:11.088033 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b938c95-c5b0-40b3-96fc-90570be05228 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:11.107908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:11.155444 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b938c95-c5b0-40b3-96fc-90570be05228 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:11.160068 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a74dc2-5b6e-452c-9bb1-c2ed8a74d403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:11.160664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a74dc2-5b6e-452c-9bb1-c2ed8a74d403 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:11.160943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a74dc2-5b6e-452c-9bb1-c2ed8a74d403 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:11.161217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a74dc2-5b6e-452c-9bb1-c2ed8a74d403 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:11.161850 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a74dc2-5b6e-452c-9bb1-c2ed8a74d403 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:11.163793 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a74dc2-5b6e-452c-9bb1-c2ed8a74d403 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:11.190385 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1424/2835] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:06:11 2022] POST /identity/v3/projects => generated 357 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:06:11.194488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed365582-94fd-4d1c-91e9-beae5db25f1f None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:11.204904 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed365582-94fd-4d1c-91e9-beae5db25f1f None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:11.264562 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2453b9bf-7f9c-45f9-a15f-3068c412dd80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:11.265196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2453b9bf-7f9c-45f9-a15f-3068c412dd80 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:11.265501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2453b9bf-7f9c-45f9-a15f-3068c412dd80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:11.265776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2453b9bf-7f9c-45f9-a15f-3068c412dd80 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:11.266444 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2453b9bf-7f9c-45f9-a15f-3068c412dd80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:11.268393 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2453b9bf-7f9c-45f9-a15f-3068c412dd80 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:11.290980 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1412/2836] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:06:11 2022] POST /identity/v3/users => generated 285 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:06:11.295662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a74dc2-5b6e-452c-9bb1-c2ed8a74d403 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:11.305710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a74dc2-5b6e-452c-9bb1-c2ed8a74d403 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:11.310409 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eefd13-249d-4ce9-826c-21faf7dd9ca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:11.310993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eefd13-249d-4ce9-826c-21faf7dd9ca2 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:11.311269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eefd13-249d-4ce9-826c-21faf7dd9ca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:11.311536 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eefd13-249d-4ce9-826c-21faf7dd9ca2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:11.312163 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26eefd13-249d-4ce9-826c-21faf7dd9ca2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:11.313742 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26eefd13-249d-4ce9-826c-21faf7dd9ca2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:11.324077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1425/2837] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:11.328677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2453b9bf-7f9c-45f9-a15f-3068c412dd80 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:11.334969 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:11.338957 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2453b9bf-7f9c-45f9-a15f-3068c412dd80 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:11.345169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a143d-8ff6-4eaf-8a95-66a1a38bd291 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:11.345811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a143d-8ff6-4eaf-8a95-66a1a38bd291 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:11.346122 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a143d-8ff6-4eaf-8a95-66a1a38bd291 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:11.346415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a143d-8ff6-4eaf-8a95-66a1a38bd291 admin admin] PATH_INFO: `/v3/projects/b5d9bcc011e74e5a8df887f81cc5568c/users/af4450c1fba6435b981d3444fa3d73f4/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:11.361312 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a143d-8ff6-4eaf-8a95-66a1a38bd291 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5d9bcc011e74e5a8df887f81cc5568c, user_id=af4450c1fba6435b981d3444fa3d73f4, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:11.364315 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a143d-8ff6-4eaf-8a95-66a1a38bd291 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:11.376878 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1413/2838] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:06:11 2022] PUT /identity/v3/projects/b5d9bcc011e74e5a8df887f81cc5568c/users/af4450c1fba6435b981d3444fa3d73f4/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:11.380764 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eefd13-249d-4ce9-826c-21faf7dd9ca2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:11.391320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eefd13-249d-4ce9-826c-21faf7dd9ca2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:11.445288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e108a51c-c5a6-414d-bdfc-8938e071499c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:11.445877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e108a51c-c5a6-414d-bdfc-8938e071499c admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:11.446181 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e108a51c-c5a6-414d-bdfc-8938e071499c admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:11.446453 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e108a51c-c5a6-414d-bdfc-8938e071499c admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:11.447080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e108a51c-c5a6-414d-bdfc-8938e071499c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:11.448662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e108a51c-c5a6-414d-bdfc-8938e071499c admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:11.458755 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1426/2839] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:11 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:11.462773 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a143d-8ff6-4eaf-8a95-66a1a38bd291 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:11.473287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a143d-8ff6-4eaf-8a95-66a1a38bd291 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:11.477990 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a710850f-4348-47e6-b298-4623a2f1e618 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:11.478673 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a710850f-4348-47e6-b298-4623a2f1e618 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:11.478949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a710850f-4348-47e6-b298-4623a2f1e618 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:11.479222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a710850f-4348-47e6-b298-4623a2f1e618 admin admin] PATH_INFO: `/v3/projects/b5d9bcc011e74e5a8df887f81cc5568c/users/af4450c1fba6435b981d3444fa3d73f4/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:11.481143 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a710850f-4348-47e6-b298-4623a2f1e618 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5d9bcc011e74e5a8df887f81cc5568c, user_id=af4450c1fba6435b981d3444fa3d73f4, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:11.483780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a710850f-4348-47e6-b298-4623a2f1e618 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:11.494170 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1414/2840] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:06:11 2022] PUT /identity/v3/projects/b5d9bcc011e74e5a8df887f81cc5568c/users/af4450c1fba6435b981d3444fa3d73f4/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:11.505413 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b015ab8-1890-443a-a6e9-6bb2d71255b9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:11.505696 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b015ab8-1890-443a-a6e9-6bb2d71255b9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:11.505966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b015ab8-1890-443a-a6e9-6bb2d71255b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:11.534136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-2b015ab8-1890-443a-a6e9-6bb2d71255b9 None None] MFA Rules not processed for user `af4450c1fba6435b981d3444fa3d73f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:11.535332 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-2b015ab8-1890-443a-a6e9-6bb2d71255b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:11.600813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1427/2841] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:11 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:11.612054 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a710850f-4348-47e6-b298-4623a2f1e618 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:11.614082 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-a710850f-4348-47e6-b298-4623a2f1e618 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:11.670612 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a710850f-4348-47e6-b298-4623a2f1e618 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:11.674218 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166cff56-d9e9-4c20-bba8-6b216141cbed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:11.674810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166cff56-d9e9-4c20-bba8-6b216141cbed service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:11.675086 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166cff56-d9e9-4c20-bba8-6b216141cbed service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:11.675360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166cff56-d9e9-4c20-bba8-6b216141cbed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:11.684025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166cff56-d9e9-4c20-bba8-6b216141cbed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:11.685431 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166cff56-d9e9-4c20-bba8-6b216141cbed service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:11.694697 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1415/2842] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:06:11 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:06:11.768698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b015ab8-1890-443a-a6e9-6bb2d71255b9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:11.778955 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b015ab8-1890-443a-a6e9-6bb2d71255b9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:11.782647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63912a9-5856-4293-a546-a3f813bf235e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:11.783232 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63912a9-5856-4293-a546-a3f813bf235e service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:11.783530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63912a9-5856-4293-a546-a3f813bf235e service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:11.783800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63912a9-5856-4293-a546-a3f813bf235e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:11.793171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:11.840678 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63912a9-5856-4293-a546-a3f813bf235e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:11.842710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63912a9-5856-4293-a546-a3f813bf235e service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:11.851506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1428/2843] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:06:11 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:06:21.690478 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=39,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:21.847458 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=22 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:26.567962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166cff56-d9e9-4c20-bba8-6b216141cbed service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:26.628111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166cff56-d9e9-4c20-bba8-6b216141cbed service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:26.632538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:26.633144 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:26.633419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:26.633688 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] PATH_INFO: `/v3/users/78004c7d5d424ed597c9d79e2a7b1f5e` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:26.634805 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78004c7d5d424ed597c9d79e2a7b1f5e)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:26.636682 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:26.682111 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78004c7d5d424ed597c9d79e2a7b1f5e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:26.682756 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78004c7d5d424ed597c9d79e2a7b1f5e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:26.690076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78004c7d5d424ed597c9d79e2a7b1f5e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:26.690714 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78004c7d5d424ed597c9d79e2a7b1f5e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:26.697317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78004c7d5d424ed597c9d79e2a7b1f5e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:26.714025 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1416/2844] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:06:26 2022] DELETE /identity/v3/users/78004c7d5d424ed597c9d79e2a7b1f5e => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:26.719211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98060f62-b35f-497a-a8fd-ede4603a09bf None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:26.719690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98060f62-b35f-497a-a8fd-ede4603a09bf None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:26.719795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98060f62-b35f-497a-a8fd-ede4603a09bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:26.739325 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-98060f62-b35f-497a-a8fd-ede4603a09bf None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:26.740513 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-98060f62-b35f-497a-a8fd-ede4603a09bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:26.797934 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1429/2845] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:26.807581 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:26.808979 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:26.864070 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a116e82-c68d-441e-9624-6cf7565d35f6 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:26.867643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972b7747-5de7-4d86-b4fa-08b2c96b1af8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:26.868234 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972b7747-5de7-4d86-b4fa-08b2c96b1af8 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:26.868508 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972b7747-5de7-4d86-b4fa-08b2c96b1af8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:26.868776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972b7747-5de7-4d86-b4fa-08b2c96b1af8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:26.877394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972b7747-5de7-4d86-b4fa-08b2c96b1af8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:26.878814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972b7747-5de7-4d86-b4fa-08b2c96b1af8 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:26.888076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1417/2846] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:06:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:06:27.143564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98060f62-b35f-497a-a8fd-ede4603a09bf None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:27.145031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-98060f62-b35f-497a-a8fd-ede4603a09bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:27.158665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98060f62-b35f-497a-a8fd-ede4603a09bf None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:27.163968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:27.164604 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:27.164880 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:27.165152 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] PATH_INFO: `/v3/projects/c9b47dd257be4ee4b11107c5ba9b734b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:27.166288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9b47dd257be4ee4b11107c5ba9b734b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:27.168189 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:27.214150 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9b47dd257be4ee4b11107c5ba9b734b', 'request_id': 'req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.221633 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9b47dd257be4ee4b11107c5ba9b734b', 'request_id': 'req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.233149 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9b47dd257be4ee4b11107c5ba9b734b', 'request_id': 'req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.245872 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] The token cache is being invalidate because project c9b47dd257be4ee4b11107c5ba9b734b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:06:27.246198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.246832 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.248856 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1430/2847] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:06:27 2022] DELETE /identity/v3/projects/c9b47dd257be4ee4b11107c5ba9b734b => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:27.252823 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972b7747-5de7-4d86-b4fa-08b2c96b1af8 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:27.262169 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972b7747-5de7-4d86-b4fa-08b2c96b1af8 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:27.279537 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:27.319028 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:27.319695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:27.319924 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:27.320198 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] PATH_INFO: `/v3/users/af4450c1fba6435b981d3444fa3d73f4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:27.321292 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af4450c1fba6435b981d3444fa3d73f4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:27.323260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:27.367550 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af4450c1fba6435b981d3444fa3d73f4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.368172 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af4450c1fba6435b981d3444fa3d73f4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.375430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af4450c1fba6435b981d3444fa3d73f4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.376043 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af4450c1fba6435b981d3444fa3d73f4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.382971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9cd665fd-99c2-424a-9de8-2a39c90077a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af4450c1fba6435b981d3444fa3d73f4'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.399811 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1418/2848] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:06:27 2022] DELETE /identity/v3/users/af4450c1fba6435b981d3444fa3d73f4 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:27.661280 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:27.662773 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:27.676545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4f3543-acd4-4bdb-89b7-5ac9e6d3f04a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:27.732614 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:27.733245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:27.733498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:27.733771 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] PATH_INFO: `/v3/projects/b5d9bcc011e74e5a8df887f81cc5568c` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:27.734914 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5d9bcc011e74e5a8df887f81cc5568c)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:27.736810 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:27.765724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:27.784196 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5d9bcc011e74e5a8df887f81cc5568c', 'request_id': 'req-5db5d56e-2c70-426a-b26a-3cf52961d696'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.790639 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5d9bcc011e74e5a8df887f81cc5568c', 'request_id': 'req-5db5d56e-2c70-426a-b26a-3cf52961d696'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.797731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5d9bcc011e74e5a8df887f81cc5568c', 'request_id': 'req-5db5d56e-2c70-426a-b26a-3cf52961d696'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.808480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] The token cache is being invalidate because project b5d9bcc011e74e5a8df887f81cc5568c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:06:27.808765 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.809402 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-5db5d56e-2c70-426a-b26a-3cf52961d696 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:27.810870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1431/2849] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:06:27 2022] DELETE /identity/v3/projects/b5d9bcc011e74e5a8df887f81cc5568c => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:27.815734 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389cd180-cd6e-4855-a117-6dc82e7646d5 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:27.816225 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389cd180-cd6e-4855-a117-6dc82e7646d5 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:27.816337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389cd180-cd6e-4855-a117-6dc82e7646d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:27.835953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-389cd180-cd6e-4855-a117-6dc82e7646d5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:27.837180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-389cd180-cd6e-4855-a117-6dc82e7646d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:27.895516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1419/2850] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:27.904317 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be8fb7-db1b-44c1-93cb-5895c0bd5489 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:27.904602 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be8fb7-db1b-44c1-93cb-5895c0bd5489 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:27.904876 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3be8fb7-db1b-44c1-93cb-5895c0bd5489 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:27.924178 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-e3be8fb7-db1b-44c1-93cb-5895c0bd5489 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:27.925383 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-e3be8fb7-db1b-44c1-93cb-5895c0bd5489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:27.933147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1432/2851] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:06:27.937354 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389cd180-cd6e-4855-a117-6dc82e7646d5 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:27.947083 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389cd180-cd6e-4855-a117-6dc82e7646d5 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:27.952108 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25cec17-0c9e-4276-b11d-58489ce2c8c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:27.952699 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25cec17-0c9e-4276-b11d-58489ce2c8c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:27.952975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25cec17-0c9e-4276-b11d-58489ce2c8c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:27.953248 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25cec17-0c9e-4276-b11d-58489ce2c8c6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:27.953888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25cec17-0c9e-4276-b11d-58489ce2c8c6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81432) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:06:27.954227 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25cec17-0c9e-4276-b11d-58489ce2c8c6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:27.955794 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25cec17-0c9e-4276-b11d-58489ce2c8c6 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:27.967521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1420/2852] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:06:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:06:27.971690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3be8fb7-db1b-44c1-93cb-5895c0bd5489 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:27.980932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3be8fb7-db1b-44c1-93cb-5895c0bd5489 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:27.985533 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19531904-1f65-473c-a8eb-52a52b5175f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:27.986147 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19531904-1f65-473c-a8eb-52a52b5175f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:27.986425 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19531904-1f65-473c-a8eb-52a52b5175f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:27.986693 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19531904-1f65-473c-a8eb-52a52b5175f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:27.987320 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19531904-1f65-473c-a8eb-52a52b5175f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:27.989464 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19531904-1f65-473c-a8eb-52a52b5175f3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.016081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1433/2853] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:06:27 2022] POST /identity/v3/projects => generated 375 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 02 01:06:28.020004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25cec17-0c9e-4276-b11d-58489ce2c8c6 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.030251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25cec17-0c9e-4276-b11d-58489ce2c8c6 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.055179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:28.087530 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6e57d8-9610-493f-b30a-b64ac8f87c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.088179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6e57d8-9610-493f-b30a-b64ac8f87c63 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.088467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6e57d8-9610-493f-b30a-b64ac8f87c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.088742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f6e57d8-9610-493f-b30a-b64ac8f87c63 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.089369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6e57d8-9610-493f-b30a-b64ac8f87c63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:28.091959 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f6e57d8-9610-493f-b30a-b64ac8f87c63 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.112702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1421/2854] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:06:28 2022] POST /identity/v3/users => generated 288 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:06:28.116885 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19531904-1f65-473c-a8eb-52a52b5175f3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.126996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19531904-1f65-473c-a8eb-52a52b5175f3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.131541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec5dc6d-9065-49f3-9742-46eead157008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.132128 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec5dc6d-9065-49f3-9742-46eead157008 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.132407 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec5dc6d-9065-49f3-9742-46eead157008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.132677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec5dc6d-9065-49f3-9742-46eead157008 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.133310 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec5dc6d-9065-49f3-9742-46eead157008 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:28.134938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec5dc6d-9065-49f3-9742-46eead157008 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.146337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1434/2855] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:28.150429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6e57d8-9610-493f-b30a-b64ac8f87c63 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.160853 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f6e57d8-9610-493f-b30a-b64ac8f87c63 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.165572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffbb2c89-f369-4b1f-a93e-51a8f61ea75e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.166221 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbb2c89-f369-4b1f-a93e-51a8f61ea75e admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.166497 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbb2c89-f369-4b1f-a93e-51a8f61ea75e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.166767 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbb2c89-f369-4b1f-a93e-51a8f61ea75e admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.167429 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffbb2c89-f369-4b1f-a93e-51a8f61ea75e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:28.168986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffbb2c89-f369-4b1f-a93e-51a8f61ea75e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.179409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1422/2856] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:28.183329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec5dc6d-9065-49f3-9742-46eead157008 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.193723 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec5dc6d-9065-49f3-9742-46eead157008 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.198382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38048f0-1fd6-40a2-a974-659d7821a41a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.199050 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38048f0-1fd6-40a2-a974-659d7821a41a admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.199329 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38048f0-1fd6-40a2-a974-659d7821a41a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.199597 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38048f0-1fd6-40a2-a974-659d7821a41a admin admin] PATH_INFO: `/v3/projects/3eaee4ae382f4c36ab29ce2f0aa96cf8/users/6f32d10fa5d340899f1656c497226d79/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.214473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38048f0-1fd6-40a2-a974-659d7821a41a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3eaee4ae382f4c36ab29ce2f0aa96cf8, user_id=6f32d10fa5d340899f1656c497226d79, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:28.217922 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38048f0-1fd6-40a2-a974-659d7821a41a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.230237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1435/2857] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:06:28 2022] PUT /identity/v3/projects/3eaee4ae382f4c36ab29ce2f0aa96cf8/users/6f32d10fa5d340899f1656c497226d79/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:28.242123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5714f4c5-4687-4e44-8eb7-be649ca2b266 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.242409 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5714f4c5-4687-4e44-8eb7-be649ca2b266 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.242679 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5714f4c5-4687-4e44-8eb7-be649ca2b266 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.271116 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-5714f4c5-4687-4e44-8eb7-be649ca2b266 None None] MFA Rules not processed for user `6f32d10fa5d340899f1656c497226d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:28.272295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5714f4c5-4687-4e44-8eb7-be649ca2b266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:28.335926 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1423/2858] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:28 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:28.340529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38048f0-1fd6-40a2-a974-659d7821a41a admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.401101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38048f0-1fd6-40a2-a974-659d7821a41a admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.405600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0d9c5a-3d1a-4582-a24c-309d253c041b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.406217 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0d9c5a-3d1a-4582-a24c-309d253c041b admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.406492 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0d9c5a-3d1a-4582-a24c-309d253c041b admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.406759 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0d9c5a-3d1a-4582-a24c-309d253c041b admin admin] PATH_INFO: `/v3/projects` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.407380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0d9c5a-3d1a-4582-a24c-309d253c041b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:28.409261 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0d9c5a-3d1a-4582-a24c-309d253c041b admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.434511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1436/2859] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:06:28 2022] POST /identity/v3/projects => generated 373 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:06:28.439258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5714f4c5-4687-4e44-8eb7-be649ca2b266 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.449849 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5714f4c5-4687-4e44-8eb7-be649ca2b266 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.504960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b522ff01-38cc-4a64-9ebd-412b470de5c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.505566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b522ff01-38cc-4a64-9ebd-412b470de5c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.505845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b522ff01-38cc-4a64-9ebd-412b470de5c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.506149 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b522ff01-38cc-4a64-9ebd-412b470de5c9 admin admin] PATH_INFO: `/v3/users` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.506802 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b522ff01-38cc-4a64-9ebd-412b470de5c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:28.508702 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b522ff01-38cc-4a64-9ebd-412b470de5c9 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.530460 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1424/2860] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:06:28 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:06:28.534613 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0d9c5a-3d1a-4582-a24c-309d253c041b admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.544554 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0d9c5a-3d1a-4582-a24c-309d253c041b admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.549097 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc290ecc-5e6c-4b08-b7fb-69b63bf18ac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.549682 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc290ecc-5e6c-4b08-b7fb-69b63bf18ac3 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.549953 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc290ecc-5e6c-4b08-b7fb-69b63bf18ac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.550259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc290ecc-5e6c-4b08-b7fb-69b63bf18ac3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.550883 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc290ecc-5e6c-4b08-b7fb-69b63bf18ac3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:28.552457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc290ecc-5e6c-4b08-b7fb-69b63bf18ac3 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.563143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1437/2861] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:28.567089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b522ff01-38cc-4a64-9ebd-412b470de5c9 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.577465 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b522ff01-38cc-4a64-9ebd-412b470de5c9 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.581983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcf165d-b7c2-4a15-8545-cdf04685f2ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.582647 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcf165d-b7c2-4a15-8545-cdf04685f2ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.582929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcf165d-b7c2-4a15-8545-cdf04685f2ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.583205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcf165d-b7c2-4a15-8545-cdf04685f2ab admin admin] PATH_INFO: `/v3/projects/6febaaa396b345e4a5056d9a2a511d64/users/4acf8047390d4e808cc91eafddb5175a/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.597428 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcf165d-b7c2-4a15-8545-cdf04685f2ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=6febaaa396b345e4a5056d9a2a511d64, user_id=4acf8047390d4e808cc91eafddb5175a, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:28.600081 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcf165d-b7c2-4a15-8545-cdf04685f2ab admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.613410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1425/2862] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:06:28 2022] PUT /identity/v3/projects/6febaaa396b345e4a5056d9a2a511d64/users/4acf8047390d4e808cc91eafddb5175a/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:28.617342 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc290ecc-5e6c-4b08-b7fb-69b63bf18ac3 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.627731 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc290ecc-5e6c-4b08-b7fb-69b63bf18ac3 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.647124 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:28.684221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d3b056-cb2a-47c2-bd11-8c42d951c425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.684772 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d3b056-cb2a-47c2-bd11-8c42d951c425 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.685638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d3b056-cb2a-47c2-bd11-8c42d951c425 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.685909 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d3b056-cb2a-47c2-bd11-8c42d951c425 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.686560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d3b056-cb2a-47c2-bd11-8c42d951c425 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:28.688137 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d3b056-cb2a-47c2-bd11-8c42d951c425 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.697673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1438/2863] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:28 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:28.701798 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcf165d-b7c2-4a15-8545-cdf04685f2ab admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.707152 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:28.713672 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcf165d-b7c2-4a15-8545-cdf04685f2ab admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.718624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88b359-76e8-43a2-a5a7-b669cfb7e5c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.719293 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88b359-76e8-43a2-a5a7-b669cfb7e5c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.719552 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88b359-76e8-43a2-a5a7-b669cfb7e5c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.719824 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be88b359-76e8-43a2-a5a7-b669cfb7e5c0 admin admin] PATH_INFO: `/v3/projects/6febaaa396b345e4a5056d9a2a511d64/users/4acf8047390d4e808cc91eafddb5175a/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.721700 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be88b359-76e8-43a2-a5a7-b669cfb7e5c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6febaaa396b345e4a5056d9a2a511d64, user_id=4acf8047390d4e808cc91eafddb5175a, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:28.724374 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be88b359-76e8-43a2-a5a7-b669cfb7e5c0 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:28.734869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1426/2864] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:06:28 2022] PUT /identity/v3/projects/6febaaa396b345e4a5056d9a2a511d64/users/4acf8047390d4e808cc91eafddb5175a/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:28.798623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c0808-c94f-4327-8b38-a132b1c4c1b7 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.799212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c0808-c94f-4327-8b38-a132b1c4c1b7 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.799212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c0808-c94f-4327-8b38-a132b1c4c1b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:28.827840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-6a8c0808-c94f-4327-8b38-a132b1c4c1b7 None None] MFA Rules not processed for user `4acf8047390d4e808cc91eafddb5175a`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:28.829043 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6a8c0808-c94f-4327-8b38-a132b1c4c1b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:28.895271 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1439/2865] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:28 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:28.905876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88b359-76e8-43a2-a5a7-b669cfb7e5c0 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:28.907675 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-be88b359-76e8-43a2-a5a7-b669cfb7e5c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:28.964295 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be88b359-76e8-43a2-a5a7-b669cfb7e5c0 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:28.968380 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd535e3-5447-4750-bdfa-abe3b1014da8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:28.968964 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd535e3-5447-4750-bdfa-abe3b1014da8 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:28.969236 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd535e3-5447-4750-bdfa-abe3b1014da8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:28.969499 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd535e3-5447-4750-bdfa-abe3b1014da8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:29.022543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd535e3-5447-4750-bdfa-abe3b1014da8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:29.023953 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd535e3-5447-4750-bdfa-abe3b1014da8 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:29.036247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1427/2866] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:06:28 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:06:31.753184 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8c0808-c94f-4327-8b38-a132b1c4c1b7 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:31.814225 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8c0808-c94f-4327-8b38-a132b1c4c1b7 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:31.818615 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:31.819227 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:31.819498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:31.819766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] PATH_INFO: `/v3/users/6f32d10fa5d340899f1656c497226d79` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:31.820862 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f32d10fa5d340899f1656c497226d79)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:31.822785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:31.867462 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f32d10fa5d340899f1656c497226d79'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:31.868616 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:31.875788 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f32d10fa5d340899f1656c497226d79'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:31.876426 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f32d10fa5d340899f1656c497226d79'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:31.877018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f32d10fa5d340899f1656c497226d79'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:31.884120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f32d10fa5d340899f1656c497226d79'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:31.901688 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1440/2867] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:06:31 2022] DELETE /identity/v3/users/6f32d10fa5d340899f1656c497226d79 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:31.906160 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edec679-1c8b-4b16-9ccd-4d3ab456c23b None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:31.907226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edec679-1c8b-4b16-9ccd-4d3ab456c23b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:31.907415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edec679-1c8b-4b16-9ccd-4d3ab456c23b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:31.926071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-5edec679-1c8b-4b16-9ccd-4d3ab456c23b None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:31.927297 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5edec679-1c8b-4b16-9ccd-4d3ab456c23b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:31.983633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1428/2868] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:31.993204 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:31.994636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:32.051697 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53253bdc-736e-455a-96ee-1ce5ea048cd2 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:32.055263 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc3d52b-b1d9-4749-a76c-7c1411661f00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:32.055863 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc3d52b-b1d9-4749-a76c-7c1411661f00 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:32.056137 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc3d52b-b1d9-4749-a76c-7c1411661f00 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:32.056405 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc3d52b-b1d9-4749-a76c-7c1411661f00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:32.065689 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc3d52b-b1d9-4749-a76c-7c1411661f00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:32.067146 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc3d52b-b1d9-4749-a76c-7c1411661f00 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:32.076335 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1441/2869] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:06:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:06:32.330253 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edec679-1c8b-4b16-9ccd-4d3ab456c23b None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:32.331742 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-5edec679-1c8b-4b16-9ccd-4d3ab456c23b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:32.344246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edec679-1c8b-4b16-9ccd-4d3ab456c23b None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:32.348777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:32.349397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:32.350190 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:32.350470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] PATH_INFO: `/v3/projects/3eaee4ae382f4c36ab29ce2f0aa96cf8` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:32.351737 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3eaee4ae382f4c36ab29ce2f0aa96cf8)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:32.353623 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:32.393023 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:32.400504 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3eaee4ae382f4c36ab29ce2f0aa96cf8', 'request_id': 'req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:32.407543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3eaee4ae382f4c36ab29ce2f0aa96cf8', 'request_id': 'req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:32.413548 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3eaee4ae382f4c36ab29ce2f0aa96cf8', 'request_id': 'req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:32.424246 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] The token cache is being invalidate because project 3eaee4ae382f4c36ab29ce2f0aa96cf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:06:32.424534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:32.425161 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:32.426644 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1429/2870] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:06:32 2022] DELETE /identity/v3/projects/3eaee4ae382f4c36ab29ce2f0aa96cf8 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:32.430638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc3d52b-b1d9-4749-a76c-7c1411661f00 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:32.440511 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc3d52b-b1d9-4749-a76c-7c1411661f00 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:32.494520 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:32.495132 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:32.495408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:32.495679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] PATH_INFO: `/v3/users/4acf8047390d4e808cc91eafddb5175a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:32.496767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] RBAC: Authorizing `identity:delete_user(user_id=4acf8047390d4e808cc91eafddb5175a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:32.498753 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:32.542459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4acf8047390d4e808cc91eafddb5175a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:32.549569 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4acf8047390d4e808cc91eafddb5175a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:32.550251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4acf8047390d4e808cc91eafddb5175a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:32.550877 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4acf8047390d4e808cc91eafddb5175a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:32.557949 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1397db27-8d21-45de-bd0f-5fa8814b935f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4acf8047390d4e808cc91eafddb5175a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:32.563681 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:32.596121 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1442/2871] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:06:32 2022] DELETE /identity/v3/users/4acf8047390d4e808cc91eafddb5175a => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:32.870572 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:32.872002 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:32.886196 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9293108e-d6c5-49fb-b02e-dd3f5921d4c5 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:32.946005 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:32.946759 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:32.946917 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:32.947187 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] PATH_INFO: `/v3/projects/6febaaa396b345e4a5056d9a2a511d64` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:32.948287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6febaaa396b345e4a5056d9a2a511d64)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:32.950194 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:32.996643 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6febaaa396b345e4a5056d9a2a511d64', 'request_id': 'req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:33.003268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6febaaa396b345e4a5056d9a2a511d64', 'request_id': 'req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:33.009379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6febaaa396b345e4a5056d9a2a511d64', 'request_id': 'req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:33.019491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] The token cache is being invalidate because project 6febaaa396b345e4a5056d9a2a511d64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:06:33.019776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:33.020398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1877c759-9a16-4ec7-b5b5-b3a29c3a7965 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:33.021842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1430/2872] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:06:32 2022] DELETE /identity/v3/projects/6febaaa396b345e4a5056d9a2a511d64 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:33.027782 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4890a492-f89d-4eed-8961-76d07807edf9 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:33.028281 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4890a492-f89d-4eed-8961-76d07807edf9 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:33.028391 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4890a492-f89d-4eed-8961-76d07807edf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:33.049288 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4890a492-f89d-4eed-8961-76d07807edf9 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:33.050457 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4890a492-f89d-4eed-8961-76d07807edf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:33.108136 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1443/2873] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:06:33.115929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d556403-f05a-4b06-b243-98d916d76e17 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:33.116214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d556403-f05a-4b06-b243-98d916d76e17 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:33.116487 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d556403-f05a-4b06-b243-98d916d76e17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:33.135437 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-7d556403-f05a-4b06-b243-98d916d76e17 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:33.136611 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7d556403-f05a-4b06-b243-98d916d76e17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:33.144755 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1431/2874] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:33.148713 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4890a492-f89d-4eed-8961-76d07807edf9 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:33.158705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4890a492-f89d-4eed-8961-76d07807edf9 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:33.163191 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63097ff-3073-4d3b-8301-bc71cc9b8b43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:33.163769 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63097ff-3073-4d3b-8301-bc71cc9b8b43 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:33.164039 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63097ff-3073-4d3b-8301-bc71cc9b8b43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:33.164307 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63097ff-3073-4d3b-8301-bc71cc9b8b43 admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:33.164938 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63097ff-3073-4d3b-8301-bc71cc9b8b43 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:06:33.165233 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63097ff-3073-4d3b-8301-bc71cc9b8b43 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:33.166865 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63097ff-3073-4d3b-8301-bc71cc9b8b43 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:33.178346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1444/2875] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:06:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:06:33.182317 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d556403-f05a-4b06-b243-98d916d76e17 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:33.191110 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d556403-f05a-4b06-b243-98d916d76e17 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:33.195729 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021eea8f-9269-4a9e-ba26-8d60df2caa83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:33.196311 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021eea8f-9269-4a9e-ba26-8d60df2caa83 admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:33.196583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021eea8f-9269-4a9e-ba26-8d60df2caa83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:33.196855 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021eea8f-9269-4a9e-ba26-8d60df2caa83 admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:33.197471 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021eea8f-9269-4a9e-ba26-8d60df2caa83 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:33.199369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021eea8f-9269-4a9e-ba26-8d60df2caa83 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:33.226903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1432/2876] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:06:33 2022] POST /identity/v3/projects => generated 391 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:06:33.234798 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63097ff-3073-4d3b-8301-bc71cc9b8b43 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:33.241410 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63097ff-3073-4d3b-8301-bc71cc9b8b43 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:33.304052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2361b86a-15ee-41c2-b12d-860e78ceb1ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:33.304572 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361b86a-15ee-41c2-b12d-860e78ceb1ae admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:33.304845 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361b86a-15ee-41c2-b12d-860e78ceb1ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:33.305111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361b86a-15ee-41c2-b12d-860e78ceb1ae admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:33.305722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2361b86a-15ee-41c2-b12d-860e78ceb1ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:33.307658 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2361b86a-15ee-41c2-b12d-860e78ceb1ae admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:33.329664 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1445/2877] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:06:33 2022] POST /identity/v3/users => generated 296 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:06:33.333780 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021eea8f-9269-4a9e-ba26-8d60df2caa83 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:33.343723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021eea8f-9269-4a9e-ba26-8d60df2caa83 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:33.348749 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d25e01-bfff-498e-8629-4bd20b185577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:33.349333 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d25e01-bfff-498e-8629-4bd20b185577 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:33.349608 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d25e01-bfff-498e-8629-4bd20b185577 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:33.349876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d25e01-bfff-498e-8629-4bd20b185577 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:33.350544 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d25e01-bfff-498e-8629-4bd20b185577 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:33.352105 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d25e01-bfff-498e-8629-4bd20b185577 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:33.361871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1433/2878] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:33.365770 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2361b86a-15ee-41c2-b12d-860e78ceb1ae admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:33.375521 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2361b86a-15ee-41c2-b12d-860e78ceb1ae admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:33.380110 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402e4697-aab8-435b-a8f2-49dd3817efc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:33.380691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402e4697-aab8-435b-a8f2-49dd3817efc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:33.380968 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402e4697-aab8-435b-a8f2-49dd3817efc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:33.381245 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402e4697-aab8-435b-a8f2-49dd3817efc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:33.381870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402e4697-aab8-435b-a8f2-49dd3817efc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:33.383477 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402e4697-aab8-435b-a8f2-49dd3817efc6 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:33.393581 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1446/2879] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:33 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:33.397425 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d25e01-bfff-498e-8629-4bd20b185577 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:33.407646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d25e01-bfff-498e-8629-4bd20b185577 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:33.412382 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd3abc1-6c5a-48ef-a8f6-0594d46c2968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:33.413013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd3abc1-6c5a-48ef-a8f6-0594d46c2968 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:33.413319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd3abc1-6c5a-48ef-a8f6-0594d46c2968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:33.413596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd3abc1-6c5a-48ef-a8f6-0594d46c2968 admin admin] PATH_INFO: `/v3/projects/e56d3617738c4751a3b4a1d0c27c56b1/users/5c5a6b0e7cd741cdb1cc47536f954088/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:33.427942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd3abc1-6c5a-48ef-a8f6-0594d46c2968 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e56d3617738c4751a3b4a1d0c27c56b1, user_id=5c5a6b0e7cd741cdb1cc47536f954088, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:33.430603 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd3abc1-6c5a-48ef-a8f6-0594d46c2968 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:33.432745 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:33.445313 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1434/2880] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:06:33 2022] PUT /identity/v3/projects/e56d3617738c4751a3b4a1d0c27c56b1/users/5c5a6b0e7cd741cdb1cc47536f954088/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:33.457210 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdacf6b-cf04-4a3c-90f1-e9c043d47d1a None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:33.457498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdacf6b-cf04-4a3c-90f1-e9c043d47d1a None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:33.457776 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdacf6b-cf04-4a3c-90f1-e9c043d47d1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:33.484717 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-4cdacf6b-cf04-4a3c-90f1-e9c043d47d1a None None] MFA Rules not processed for user `5c5a6b0e7cd741cdb1cc47536f954088`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:33.485908 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-4cdacf6b-cf04-4a3c-90f1-e9c043d47d1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:33.527962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:33.553537 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1447/2881] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:33 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:33.564846 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd3abc1-6c5a-48ef-a8f6-0594d46c2968 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:33.566542 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-1fd3abc1-6c5a-48ef-a8f6-0594d46c2968 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:33.623230 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd3abc1-6c5a-48ef-a8f6-0594d46c2968 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:33.627385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f81ca7-aac3-4548-9379-94f19c11ea96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:33.627983 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f81ca7-aac3-4548-9379-94f19c11ea96 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:33.628262 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f81ca7-aac3-4548-9379-94f19c11ea96 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:33.628534 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f81ca7-aac3-4548-9379-94f19c11ea96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:33.636474 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f81ca7-aac3-4548-9379-94f19c11ea96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:33.637912 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f81ca7-aac3-4548-9379-94f19c11ea96 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:33.647135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1435/2882] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:06:33 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:06:43.550628 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=8 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:43.643159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:INSERT=1,keystone:SELECT=28 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:51.889258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdacf6b-cf04-4a3c-90f1-e9c043d47d1a None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:51.949725 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdacf6b-cf04-4a3c-90f1-e9c043d47d1a None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:51.954173 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:51.955284 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:51.955558 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:51.955826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] PATH_INFO: `/v3/users/5c5a6b0e7cd741cdb1cc47536f954088` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:51.956984 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c5a6b0e7cd741cdb1cc47536f954088)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:51.958920 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:52.004082 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c5a6b0e7cd741cdb1cc47536f954088'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:52.010739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c5a6b0e7cd741cdb1cc47536f954088'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:52.011380 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c5a6b0e7cd741cdb1cc47536f954088'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:52.011974 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c5a6b0e7cd741cdb1cc47536f954088'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:52.019304 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c5a6b0e7cd741cdb1cc47536f954088'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:52.037237 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1448/2883] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:06:51 2022] DELETE /identity/v3/users/5c5a6b0e7cd741cdb1cc47536f954088 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:52.041826 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1241c90-bd12-4687-b92a-ab5de617f795 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.042419 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1241c90-bd12-4687-b92a-ab5de617f795 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.042524 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1241c90-bd12-4687-b92a-ab5de617f795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.061684 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-c1241c90-bd12-4687-b92a-ab5de617f795 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:52.062901 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c1241c90-bd12-4687-b92a-ab5de617f795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:52.119168 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1436/2884] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:52.129258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:52.130647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:52.185274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd29da6-807e-465e-aad0-2c5fe95203db admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:52.189355 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15e4cd6-fba4-432c-8e99-f140a87082b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:52.189939 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e4cd6-fba4-432c-8e99-f140a87082b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.190248 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e4cd6-fba4-432c-8e99-f140a87082b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.190517 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15e4cd6-fba4-432c-8e99-f140a87082b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.199206 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15e4cd6-fba4-432c-8e99-f140a87082b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:52.200621 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15e4cd6-fba4-432c-8e99-f140a87082b9 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:52.210358 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1449/2885] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:06:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:06:52.467902 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1241c90-bd12-4687-b92a-ab5de617f795 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:52.469369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c1241c90-bd12-4687-b92a-ab5de617f795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:52.481016 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1241c90-bd12-4687-b92a-ab5de617f795 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:52.485616 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:52.486268 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.486543 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.486810 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] PATH_INFO: `/v3/projects/e56d3617738c4751a3b4a1d0c27c56b1` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.488174 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e56d3617738c4751a3b4a1d0c27c56b1)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:52.490069 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:52.536851 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e56d3617738c4751a3b4a1d0c27c56b1', 'request_id': 'req-1a8106df-5707-47f9-bb28-baac8a41396e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:52.545000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e56d3617738c4751a3b4a1d0c27c56b1', 'request_id': 'req-1a8106df-5707-47f9-bb28-baac8a41396e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:52.551625 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e56d3617738c4751a3b4a1d0c27c56b1', 'request_id': 'req-1a8106df-5707-47f9-bb28-baac8a41396e'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:52.561943 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] The token cache is being invalidate because project e56d3617738c4751a3b4a1d0c27c56b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81432) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:06:52.562257 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:52.562995 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-1a8106df-5707-47f9-bb28-baac8a41396e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:06:52.565050 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1437/2886] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:06:52 2022] DELETE /identity/v3/projects/e56d3617738c4751a3b4a1d0c27c56b1 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:52.569921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae7192-2e41-4406-8893-c54600ff2b50 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.570236 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae7192-2e41-4406-8893-c54600ff2b50 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.570540 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae7192-2e41-4406-8893-c54600ff2b50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.590223 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-f1ae7192-2e41-4406-8893-c54600ff2b50 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:52.591432 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f1ae7192-2e41-4406-8893-c54600ff2b50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:52.612031 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:52.654142 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1450/2887] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:52.661697 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c27174-1b21-4719-8b93-6a58644b1235 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.661978 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c27174-1b21-4719-8b93-6a58644b1235 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.662273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c27174-1b21-4719-8b93-6a58644b1235 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.681941 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-85c27174-1b21-4719-8b93-6a58644b1235 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:52.683150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-85c27174-1b21-4719-8b93-6a58644b1235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:52.691379 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1438/2888] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:52.695322 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ae7192-2e41-4406-8893-c54600ff2b50 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:52.704695 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ae7192-2e41-4406-8893-c54600ff2b50 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:52.709228 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6027e03e-c09e-4686-a1de-6a5fd35c0efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:52.709813 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6027e03e-c09e-4686-a1de-6a5fd35c0efb admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.710120 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6027e03e-c09e-4686-a1de-6a5fd35c0efb admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.710398 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6027e03e-c09e-4686-a1de-6a5fd35c0efb admin admin] PATH_INFO: `/v3/domains` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.711036 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6027e03e-c09e-4686-a1de-6a5fd35c0efb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81433) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 02 01:06:52.711337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6027e03e-c09e-4686-a1de-6a5fd35c0efb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:52.712924 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6027e03e-c09e-4686-a1de-6a5fd35c0efb admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:52.724101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1451/2889] 10.222.0.25 () {58 vars in 1153 bytes} [Sun Oct 2 01:06:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 02 01:06:52.728153 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c27174-1b21-4719-8b93-6a58644b1235 None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:52.737427 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c27174-1b21-4719-8b93-6a58644b1235 None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:52.742276 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e8eb23-83e8-40ca-ab78-10f2cf5c3b3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:52.742852 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e8eb23-83e8-40ca-ab78-10f2cf5c3b3b admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.743123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e8eb23-83e8-40ca-ab78-10f2cf5c3b3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.743389 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e8eb23-83e8-40ca-ab78-10f2cf5c3b3b admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.744000 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e8eb23-83e8-40ca-ab78-10f2cf5c3b3b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:52.745858 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e8eb23-83e8-40ca-ab78-10f2cf5c3b3b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:52.772569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1439/2890] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:06:52 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:06:52.776635 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6027e03e-c09e-4686-a1de-6a5fd35c0efb admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:52.786620 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6027e03e-c09e-4686-a1de-6a5fd35c0efb admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:52.841889 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8568e4db-2d00-4563-ad63-61f0e45c6cfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:52.842650 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8568e4db-2d00-4563-ad63-61f0e45c6cfe admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.842929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8568e4db-2d00-4563-ad63-61f0e45c6cfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.843198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8568e4db-2d00-4563-ad63-61f0e45c6cfe admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.843826 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8568e4db-2d00-4563-ad63-61f0e45c6cfe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:52.845785 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8568e4db-2d00-4563-ad63-61f0e45c6cfe admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:52.867463 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1452/2891] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:06:52 2022] POST /identity/v3/users => generated 292 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:06:52.870829 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e8eb23-83e8-40ca-ab78-10f2cf5c3b3b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:52.881388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e8eb23-83e8-40ca-ab78-10f2cf5c3b3b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:52.886079 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d714deff-6447-4003-a986-7ea6b16c2c8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:52.886740 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d714deff-6447-4003-a986-7ea6b16c2c8d admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.887022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d714deff-6447-4003-a986-7ea6b16c2c8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.887320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d714deff-6447-4003-a986-7ea6b16c2c8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.888009 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d714deff-6447-4003-a986-7ea6b16c2c8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:52.889578 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d714deff-6447-4003-a986-7ea6b16c2c8d admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:52.899344 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1440/2892] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:52.903168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8568e4db-2d00-4563-ad63-61f0e45c6cfe admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:52.913560 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8568e4db-2d00-4563-ad63-61f0e45c6cfe admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:52.918081 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a97036-9a17-4982-aca1-ca07a9b5bc87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:52.918671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a97036-9a17-4982-aca1-ca07a9b5bc87 admin admin] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.918946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a97036-9a17-4982-aca1-ca07a9b5bc87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.919216 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a97036-9a17-4982-aca1-ca07a9b5bc87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.919840 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a97036-9a17-4982-aca1-ca07a9b5bc87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:52.921412 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a97036-9a17-4982-aca1-ca07a9b5bc87 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:52.932481 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1453/2893] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:52.936633 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d714deff-6447-4003-a986-7ea6b16c2c8d admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:52.947261 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d714deff-6447-4003-a986-7ea6b16c2c8d admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:52.951776 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce246ea-0430-44b5-b6ee-de51091c3842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:52.952412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce246ea-0430-44b5-b6ee-de51091c3842 admin admin] REQUEST_METHOD: `PUT` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.952687 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce246ea-0430-44b5-b6ee-de51091c3842 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.952958 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce246ea-0430-44b5-b6ee-de51091c3842 admin admin] PATH_INFO: `/v3/projects/ff4a21c5f4b04110b2fa42a2bba32348/users/47e0ae04969b4f7684cb21edaaa85b55/roles/e4fa1b2d8c404336a855b6142fbf25d4` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:52.967267 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce246ea-0430-44b5-b6ee-de51091c3842 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff4a21c5f4b04110b2fa42a2bba32348, user_id=47e0ae04969b4f7684cb21edaaa85b55, role_id=e4fa1b2d8c404336a855b6142fbf25d4)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:52.969888 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce246ea-0430-44b5-b6ee-de51091c3842 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:52.982935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1441/2894] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:06:52 2022] PUT /identity/v3/projects/ff4a21c5f4b04110b2fa42a2bba32348/users/47e0ae04969b4f7684cb21edaaa85b55/roles/e4fa1b2d8c404336a855b6142fbf25d4 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:52.988733 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53efc2b8-e0ef-49a6-93e4-6fd8f1d611c5 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:52.989041 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53efc2b8-e0ef-49a6-93e4-6fd8f1d611c5 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:52.989315 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53efc2b8-e0ef-49a6-93e4-6fd8f1d611c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:53.007545 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-53efc2b8-e0ef-49a6-93e4-6fd8f1d611c5 None None] MFA Rules not processed for user `b3c5f8b9f24f40b99d0e28c5d77526fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:53.008722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-53efc2b8-e0ef-49a6-93e4-6fd8f1d611c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:53.067331 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1454/2895] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:06:53.076968 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce246ea-0430-44b5-b6ee-de51091c3842 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:53.078703 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3ce246ea-0430-44b5-b6ee-de51091c3842 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:53.100490 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:53.138180 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce246ea-0430-44b5-b6ee-de51091c3842 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:53.141873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490f00fb-2aed-4cc5-8f5e-f82ec0612426 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:53.142498 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490f00fb-2aed-4cc5-8f5e-f82ec0612426 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:53.142777 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490f00fb-2aed-4cc5-8f5e-f82ec0612426 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:53.143049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490f00fb-2aed-4cc5-8f5e-f82ec0612426 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:53.151746 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490f00fb-2aed-4cc5-8f5e-f82ec0612426 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:53.153159 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490f00fb-2aed-4cc5-8f5e-f82ec0612426 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:53.162976 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1442/2896] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:06:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:06:57.055600 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f5dc19-8575-411e-a540-2e6b410bbcd2 None None] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:57.056063 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f5dc19-8575-411e-a540-2e6b410bbcd2 None None] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:57.056179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f5dc19-8575-411e-a540-2e6b410bbcd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:57.097812 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.auth.core [None req-a1f5dc19-8575-411e-a540-2e6b410bbcd2 None None] MFA Rules not processed for user `47e0ae04969b4f7684cb21edaaa85b55`. Rule list: `[]` (Enabled: `True`). {{(pid=81433) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:06:57.099299 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a1f5dc19-8575-411e-a540-2e6b410bbcd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:06:57.127438 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:06:57.189549 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1455/2897] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:06:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 02 01:06:57.193594 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490f00fb-2aed-4cc5-8f5e-f82ec0612426 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:57.203576 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490f00fb-2aed-4cc5-8f5e-f82ec0612426 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:57.208677 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0f25d9-778f-4fd2-80f5-35314d7266dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:57.209398 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0f25d9-778f-4fd2-80f5-35314d7266dc admin admin] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:57.209874 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0f25d9-778f-4fd2-80f5-35314d7266dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:57.210589 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0f25d9-778f-4fd2-80f5-35314d7266dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:57.211438 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0f25d9-778f-4fd2-80f5-35314d7266dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:57.213435 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0f25d9-778f-4fd2-80f5-35314d7266dc admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:57.248157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1443/2898] 10.222.0.25 () {60 vars in 1153 bytes} [Sun Oct 2 01:06:57 2022] POST /identity/v3/projects => generated 385 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:06:57.253133 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f5dc19-8575-411e-a540-2e6b410bbcd2 None None] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:57.264408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f5dc19-8575-411e-a540-2e6b410bbcd2 None None] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:57.323860 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db17d911-f9f6-4f8d-a4df-f42431f21f53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:57.324471 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db17d911-f9f6-4f8d-a4df-f42431f21f53 admin admin] REQUEST_METHOD: `POST` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:57.324751 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db17d911-f9f6-4f8d-a4df-f42431f21f53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:57.325026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db17d911-f9f6-4f8d-a4df-f42431f21f53 admin admin] PATH_INFO: `/v3/users` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:57.326145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db17d911-f9f6-4f8d-a4df-f42431f21f53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:57.328348 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db17d911-f9f6-4f8d-a4df-f42431f21f53 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:57.350646 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1456/2899] 10.222.0.25 () {60 vars in 1144 bytes} [Sun Oct 2 01:06:57 2022] POST /identity/v3/users => generated 299 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 02 01:06:57.354469 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0f25d9-778f-4fd2-80f5-35314d7266dc admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:57.365375 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0f25d9-778f-4fd2-80f5-35314d7266dc admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:57.369971 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9e55bf-42b1-496d-a9f9-dbdc0eeb15c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:57.370600 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e55bf-42b1-496d-a9f9-dbdc0eeb15c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:57.370881 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e55bf-42b1-496d-a9f9-dbdc0eeb15c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:57.371156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9e55bf-42b1-496d-a9f9-dbdc0eeb15c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:57.371795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9e55bf-42b1-496d-a9f9-dbdc0eeb15c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:57.373373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9e55bf-42b1-496d-a9f9-dbdc0eeb15c5 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:57.391392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1444/2900] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:57 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:57.395691 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db17d911-f9f6-4f8d-a4df-f42431f21f53 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:57.405841 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db17d911-f9f6-4f8d-a4df-f42431f21f53 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:57.411003 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa0c973-b0c7-4f05-a074-c32b2c4a54f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:57.411649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa0c973-b0c7-4f05-a074-c32b2c4a54f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:57.411923 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa0c973-b0c7-4f05-a074-c32b2c4a54f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:57.412188 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa0c973-b0c7-4f05-a074-c32b2c4a54f8 admin admin] PATH_INFO: `/v3/projects/e64f998af7b54e848da66c621311e42b/users/b0c55d58b8c040479b98026cdb535e07/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:57.428665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa0c973-b0c7-4f05-a074-c32b2c4a54f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e64f998af7b54e848da66c621311e42b, user_id=b0c55d58b8c040479b98026cdb535e07, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:57.431368 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa0c973-b0c7-4f05-a074-c32b2c4a54f8 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:57.449129 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1457/2901] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:06:57 2022] PUT /identity/v3/projects/e64f998af7b54e848da66c621311e42b/users/b0c55d58b8c040479b98026cdb535e07/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:06:57.453143 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9e55bf-42b1-496d-a9f9-dbdc0eeb15c5 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:57.463377 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9e55bf-42b1-496d-a9f9-dbdc0eeb15c5 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:57.521189 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fb48fa-829f-4cc4-84c9-1f1ea82804f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:57.521790 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fb48fa-829f-4cc4-84c9-1f1ea82804f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:57.522076 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fb48fa-829f-4cc4-84c9-1f1ea82804f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:57.522358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fb48fa-829f-4cc4-84c9-1f1ea82804f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:57.522986 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fb48fa-829f-4cc4-84c9-1f1ea82804f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:57.524580 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fb48fa-829f-4cc4-84c9-1f1ea82804f4 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:57.534671 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1445/2902] 10.222.0.25 () {58 vars in 1122 bytes} [Sun Oct 2 01:06:57 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 02 01:06:57.538817 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa0c973-b0c7-4f05-a074-c32b2c4a54f8 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:06:57.549652 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa0c973-b0c7-4f05-a074-c32b2c4a54f8 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:06:57.554171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63798f4-69ad-486b-89ca-ca52c7a6c7c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:06:57.554808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63798f4-69ad-486b-89ca-ca52c7a6c7c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:06:57.555079 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63798f4-69ad-486b-89ca-ca52c7a6c7c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:06:57.555345 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63798f4-69ad-486b-89ca-ca52c7a6c7c1 admin admin] PATH_INFO: `/v3/projects/e64f998af7b54e848da66c621311e42b/users/b0c55d58b8c040479b98026cdb535e07/roles/e6c41ef419da4c0d98f540b23c2e1a8a` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:06:57.557214 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63798f4-69ad-486b-89ca-ca52c7a6c7c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e64f998af7b54e848da66c621311e42b, user_id=b0c55d58b8c040479b98026cdb535e07, role_id=e6c41ef419da4c0d98f540b23c2e1a8a)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:06:57.559864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63798f4-69ad-486b-89ca-ca52c7a6c7c1 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:06:57.569789 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1458/2903] 10.222.0.25 () {60 vars in 1483 bytes} [Sun Oct 2 01:06:57 2022] PUT /identity/v3/projects/e64f998af7b54e848da66c621311e42b/users/b0c55d58b8c040479b98026cdb535e07/roles/e6c41ef419da4c0d98f540b23c2e1a8a => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 02 01:07:01.692247 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384c5d95-2e4e-457b-b130-307f35091ad9 None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:07:01.693222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384c5d95-2e4e-457b-b130-307f35091ad9 None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:07:01.693949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384c5d95-2e4e-457b-b130-307f35091ad9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:07:01.756987 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-384c5d95-2e4e-457b-b130-307f35091ad9 None None] MFA Rules not processed for user `b0c55d58b8c040479b98026cdb535e07`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:07:01.758388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-384c5d95-2e4e-457b-b130-307f35091ad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:07:01.878027 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1446/2904] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:07:01 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:07:01.902571 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63798f4-69ad-486b-89ca-ca52c7a6c7c1 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:07:01.904527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c63798f4-69ad-486b-89ca-ca52c7a6c7c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:07:01.999874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63798f4-69ad-486b-89ca-ca52c7a6c7c1 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:07:02.006722 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27869cd7-16d6-4a11-bca6-9ad591ae8d09 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30fc59f80f544cef8b3869e8d37bb341', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:07:02.007377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27869cd7-16d6-4a11-bca6-9ad591ae8d09 service glance] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:07:02.007654 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27869cd7-16d6-4a11-bca6-9ad591ae8d09 service glance] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:07:02.008066 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27869cd7-16d6-4a11-bca6-9ad591ae8d09 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:07:02.012673 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:07:02.074582 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27869cd7-16d6-4a11-bca6-9ad591ae8d09 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:07:02.076145 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27869cd7-16d6-4a11-bca6-9ad591ae8d09 service glance] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:07:02.086970 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1459/2905] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:07:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:07:02.227486 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48746236-06b8-4ecc-97ea-c96ad98d032b None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:07:02.227778 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48746236-06b8-4ecc-97ea-c96ad98d032b None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:07:02.228062 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48746236-06b8-4ecc-97ea-c96ad98d032b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:07:02.246470 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:07:02.296065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-48746236-06b8-4ecc-97ea-c96ad98d032b None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:07:02.297320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-48746236-06b8-4ecc-97ea-c96ad98d032b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:07:02.366199 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1447/2906] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 01:07:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:07:02.412710 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27869cd7-16d6-4a11-bca6-9ad591ae8d09 service glance] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:07:02.414116 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-27869cd7-16d6-4a11-bca6-9ad591ae8d09 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:07:02.508055 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27869cd7-16d6-4a11-bca6-9ad591ae8d09 service glance] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:07:02.513030 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa75189a-404c-45d4-9986-35613f4687c4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:07:02.513623 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa75189a-404c-45d4-9986-35613f4687c4 service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:07:02.513943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa75189a-404c-45d4-9986-35613f4687c4 service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:07:02.515505 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa75189a-404c-45d4-9986-35613f4687c4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:07:02.523155 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa75189a-404c-45d4-9986-35613f4687c4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:07:02.524730 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa75189a-404c-45d4-9986-35613f4687c4 service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:07:02.532912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1460/2907] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 01:07:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 01:07:11.180585 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98fa94c-ce1b-4ef6-b847-114c0060cecc None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:07:11.181077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98fa94c-ce1b-4ef6-b847-114c0060cecc None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:07:11.181176 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98fa94c-ce1b-4ef6-b847-114c0060cecc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:07:11.199980 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f98fa94c-ce1b-4ef6-b847-114c0060cecc None None] MFA Rules not processed for user `47e0ae04969b4f7684cb21edaaa85b55`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:07:11.201157 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f98fa94c-ce1b-4ef6-b847-114c0060cecc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:07:11.208583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1448/2908] 10.222.0.25 () {58 vars in 958 bytes} [Sun Oct 2 01:07:11 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:07:11.473052 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa75189a-404c-45d4-9986-35613f4687c4 service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:07:11.474491 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-aa75189a-404c-45d4-9986-35613f4687c4 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:07:11.529416 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa75189a-404c-45d4-9986-35613f4687c4 service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:07:11.532965 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcec58c-b306-4c6f-9985-d8e9e3fa240c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:07:11.533544 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcec58c-b306-4c6f-9985-d8e9e3fa240c service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:07:11.533815 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcec58c-b306-4c6f-9985-d8e9e3fa240c service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:07:11.534117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edcec58c-b306-4c6f-9985-d8e9e3fa240c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:07:11.535760 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-edcec58c-b306-4c6f-9985-d8e9e3fa240c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:07:11.544799 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcec58c-b306-4c6f-9985-d8e9e3fa240c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:07:11.546231 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edcec58c-b306-4c6f-9985-d8e9e3fa240c service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:07:11.554586 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1461/2909] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:07:11 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:07:13.395410 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03430fb-736a-480d-9b02-a90b9b0e60cd None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:07:13.395890 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03430fb-736a-480d-9b02-a90b9b0e60cd None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:07:13.396004 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03430fb-736a-480d-9b02-a90b9b0e60cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:07:13.416907 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-f03430fb-736a-480d-9b02-a90b9b0e60cd None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:07:13.418363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f03430fb-736a-480d-9b02-a90b9b0e60cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:07:13.426583 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1449/2910] 10.222.0.25 () {58 vars in 1009 bytes} [Sun Oct 2 01:07:13 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:07:13.438506 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcec58c-b306-4c6f-9985-d8e9e3fa240c service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:07:13.450346 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edcec58c-b306-4c6f-9985-d8e9e3fa240c service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:07:13.455251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904127f1-c0be-44e8-962e-ab539c5fc38b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:07:13.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904127f1-c0be-44e8-962e-ab539c5fc38b service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:07:13.456117 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904127f1-c0be-44e8-962e-ab539c5fc38b service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:07:13.456382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904127f1-c0be-44e8-962e-ab539c5fc38b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:07:13.465367 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904127f1-c0be-44e8-962e-ab539c5fc38b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:07:13.467198 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904127f1-c0be-44e8-962e-ab539c5fc38b service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:07:13.475834 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1462/2911] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 01:07:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 02 01:07:23.411243 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=45 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:07:23.472525 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=98 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:07:56.353974 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03430fb-736a-480d-9b02-a90b9b0e60cd None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:07:56.355378 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-f03430fb-736a-480d-9b02-a90b9b0e60cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:07:56.410397 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03430fb-736a-480d-9b02-a90b9b0e60cd None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:07:56.413965 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829db7fb-8879-4775-969c-b32414b065a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:07:56.414597 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829db7fb-8879-4775-969c-b32414b065a1 service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:07:56.414876 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829db7fb-8879-4775-969c-b32414b065a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:07:56.415150 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829db7fb-8879-4775-969c-b32414b065a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:07:56.416819 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-829db7fb-8879-4775-969c-b32414b065a1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:07:56.472748 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829db7fb-8879-4775-969c-b32414b065a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:07:56.474215 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829db7fb-8879-4775-969c-b32414b065a1 service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:07:56.482935 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1450/2912] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:07:56 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:08:06.478948 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=48 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:08:28.354596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904127f1-c0be-44e8-962e-ab539c5fc38b service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:08:28.356010 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-904127f1-c0be-44e8-962e-ab539c5fc38b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:08:28.428338 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904127f1-c0be-44e8-962e-ab539c5fc38b service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:08:28.433486 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8274f814-2248-4c79-80fd-e48a088463f6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:08:28.434254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8274f814-2248-4c79-80fd-e48a088463f6 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:08:28.434530 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8274f814-2248-4c79-80fd-e48a088463f6 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:08:28.434800 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8274f814-2248-4c79-80fd-e48a088463f6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:08:28.436502 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8274f814-2248-4c79-80fd-e48a088463f6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:08:28.448828 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8274f814-2248-4c79-80fd-e48a088463f6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:08:28.450433 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8274f814-2248-4c79-80fd-e48a088463f6 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:08:28.460472 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1463/2913] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:08:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:08:28.583860 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829db7fb-8879-4775-969c-b32414b065a1 service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:08:28.593406 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829db7fb-8879-4775-969c-b32414b065a1 service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:08:28.596998 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33634b6b-1997-44b3-896d-dab35ed629c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:08:28.597596 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33634b6b-1997-44b3-896d-dab35ed629c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:08:28.598078 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33634b6b-1997-44b3-896d-dab35ed629c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:08:28.598212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33634b6b-1997-44b3-896d-dab35ed629c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:08:28.607630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33634b6b-1997-44b3-896d-dab35ed629c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:08:28.609071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33634b6b-1997-44b3-896d-dab35ed629c8 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:08:28.619071 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1451/2914] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:08:28 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:08:38.456002 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=32 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:08:38.615538 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:10:55.959354 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8274f814-2248-4c79-80fd-e48a088463f6 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:10:55.969264 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8274f814-2248-4c79-80fd-e48a088463f6 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:10:55.972803 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96bfd3a-89fe-4558-9735-c1713b4d32ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:10:55.973388 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96bfd3a-89fe-4558-9735-c1713b4d32ba service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:10:55.973662 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96bfd3a-89fe-4558-9735-c1713b4d32ba service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:10:55.973929 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96bfd3a-89fe-4558-9735-c1713b4d32ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:10:55.975655 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a96bfd3a-89fe-4558-9735-c1713b4d32ba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:10:55.985500 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96bfd3a-89fe-4558-9735-c1713b4d32ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:10:55.986940 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96bfd3a-89fe-4558-9735-c1713b4d32ba service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:10:55.994981 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1464/2915] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:10:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:11:05.991456 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:12:01.862342 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33634b6b-1997-44b3-896d-dab35ed629c8 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:12:01.863844 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-33634b6b-1997-44b3-896d-dab35ed629c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:12:01.935260 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33634b6b-1997-44b3-896d-dab35ed629c8 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:12:01.941302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bb2e4f-97d3-4df0-a1fa-07c8935fec2c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:12:01.941895 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bb2e4f-97d3-4df0-a1fa-07c8935fec2c service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:12:01.942217 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bb2e4f-97d3-4df0-a1fa-07c8935fec2c service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:12:01.942501 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bb2e4f-97d3-4df0-a1fa-07c8935fec2c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:12:01.950388 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bb2e4f-97d3-4df0-a1fa-07c8935fec2c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:12:01.951988 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bb2e4f-97d3-4df0-a1fa-07c8935fec2c service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:12:01.960774 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1452/2916] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:12:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:12:11.956817 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=34 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:12:56.741763 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96bfd3a-89fe-4558-9735-c1713b4d32ba service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:12:56.743212 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-a96bfd3a-89fe-4558-9735-c1713b4d32ba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:12:56.785996 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96bfd3a-89fe-4558-9735-c1713b4d32ba service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:12:56.789739 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68968f35-8981-421b-902f-4d258e83c46c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:12:56.790362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68968f35-8981-421b-902f-4d258e83c46c service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:12:56.790636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68968f35-8981-421b-902f-4d258e83c46c service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:12:56.790905 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68968f35-8981-421b-902f-4d258e83c46c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:12:56.811708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68968f35-8981-421b-902f-4d258e83c46c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:12:56.813105 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68968f35-8981-421b-902f-4d258e83c46c service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:12:56.839282 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1465/2917] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:12:56 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:13:06.836822 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=32 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:13:59.989965 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bb2e4f-97d3-4df0-a1fa-07c8935fec2c service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:14:00.040917 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bb2e4f-97d3-4df0-a1fa-07c8935fec2c service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:14:00.046032 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759f42ec-19bd-43e1-bc9c-7d4fadafa710 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:14:00.046662 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759f42ec-19bd-43e1-bc9c-7d4fadafa710 service placement] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:14:00.046942 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759f42ec-19bd-43e1-bc9c-7d4fadafa710 service placement] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:14:00.047214 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759f42ec-19bd-43e1-bc9c-7d4fadafa710 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:14:00.055781 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759f42ec-19bd-43e1-bc9c-7d4fadafa710 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:14:00.057341 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759f42ec-19bd-43e1-bc9c-7d4fadafa710 service placement] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:14:00.067519 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1453/2918] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:13:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:14:10.063302 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=24 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:15:31.766874 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68968f35-8981-421b-902f-4d258e83c46c service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:15:31.784708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68968f35-8981-421b-902f-4d258e83c46c service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:15:31.788254 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4d5ea7-8b6a-4462-b4ff-f2f360d2a21f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:15:31.788831 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4d5ea7-8b6a-4462-b4ff-f2f360d2a21f service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:15:31.789101 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4d5ea7-8b6a-4462-b4ff-f2f360d2a21f service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:15:31.789366 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4d5ea7-8b6a-4462-b4ff-f2f360d2a21f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:15:31.791132 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-1b4d5ea7-8b6a-4462-b4ff-f2f360d2a21f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:15:31.823193 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4d5ea7-8b6a-4462-b4ff-f2f360d2a21f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:15:31.824564 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4d5ea7-8b6a-4462-b4ff-f2f360d2a21f service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:15:31.844576 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1466/2919] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:15:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:15:41.841704 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=22 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:16:02.851390 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759f42ec-19bd-43e1-bc9c-7d4fadafa710 service placement] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:16:02.870832 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759f42ec-19bd-43e1-bc9c-7d4fadafa710 service placement] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:16:02.874430 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22f9528-6a8a-4d8a-b3f2-f8f8d3e62764 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:16:02.875022 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22f9528-6a8a-4d8a-b3f2-f8f8d3e62764 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:16:02.875298 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22f9528-6a8a-4d8a-b3f2-f8f8d3e62764 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:16:02.875566 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22f9528-6a8a-4d8a-b3f2-f8f8d3e62764 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:16:02.882661 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22f9528-6a8a-4d8a-b3f2-f8f8d3e62764 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:16:02.884074 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22f9528-6a8a-4d8a-b3f2-f8f8d3e62764 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:16:02.891821 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1454/2920] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:16:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:16:12.888327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=10 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:17:02.052220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4d5ea7-8b6a-4462-b4ff-f2f360d2a21f service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:17:02.060937 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4d5ea7-8b6a-4462-b4ff-f2f360d2a21f service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:17:02.064566 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f329a233-baad-422f-8ae8-fb936722c523 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:17:02.065171 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f329a233-baad-422f-8ae8-fb936722c523 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:17:02.065440 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f329a233-baad-422f-8ae8-fb936722c523 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:17:02.065708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f329a233-baad-422f-8ae8-fb936722c523 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:17:02.067565 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f329a233-baad-422f-8ae8-fb936722c523 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:17:02.149143 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f329a233-baad-422f-8ae8-fb936722c523 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:17:02.150667 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f329a233-baad-422f-8ae8-fb936722c523 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:17:02.169864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1467/2921] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:17:02 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:17:12.166638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=41 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:17:57.204251 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22f9528-6a8a-4d8a-b3f2-f8f8d3e62764 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:17:57.205845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-e22f9528-6a8a-4d8a-b3f2-f8f8d3e62764 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:17:57.273521 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22f9528-6a8a-4d8a-b3f2-f8f8d3e62764 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:17:57.277414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa2f069-93aa-42f5-a828-cee14a7f4f7f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:17:57.278049 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa2f069-93aa-42f5-a828-cee14a7f4f7f service cinder] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:17:57.278344 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa2f069-93aa-42f5-a828-cee14a7f4f7f service cinder] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:17:57.278614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa2f069-93aa-42f5-a828-cee14a7f4f7f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:17:57.280457 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-6fa2f069-93aa-42f5-a828-cee14a7f4f7f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:17:57.348949 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa2f069-93aa-42f5-a828-cee14a7f4f7f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:17:57.350414 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa2f069-93aa-42f5-a828-cee14a7f4f7f service cinder] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:17:57.359592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1455/2922] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:17:57 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:18:07.355642 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=58 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:19:05.980659 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f329a233-baad-422f-8ae8-fb936722c523 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:19:05.982269 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-f329a233-baad-422f-8ae8-fb936722c523 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:19:06.062274 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f329a233-baad-422f-8ae8-fb936722c523 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:19:06.067349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31336373-ed5d-45d4-9924-27b10f179556 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:19:06.067980 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31336373-ed5d-45d4-9924-27b10f179556 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:19:06.068259 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31336373-ed5d-45d4-9924-27b10f179556 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:19:06.068529 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31336373-ed5d-45d4-9924-27b10f179556 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:19:06.070993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-31336373-ed5d-45d4-9924-27b10f179556 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:19:06.082297 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31336373-ed5d-45d4-9924-27b10f179556 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:19:06.083864 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31336373-ed5d-45d4-9924-27b10f179556 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:19:06.093596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1468/2923] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:19:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:19:16.089461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=37 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:19:40.415869 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa2f069-93aa-42f5-a828-cee14a7f4f7f service cinder] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:19:40.430415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa2f069-93aa-42f5-a828-cee14a7f4f7f service cinder] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:19:40.436376 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a5466b-0c3d-4e82-b579-fcc7ec074039 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:19:40.437061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a5466b-0c3d-4e82-b579-fcc7ec074039 service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:19:40.437412 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a5466b-0c3d-4e82-b579-fcc7ec074039 service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:19:40.437762 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a5466b-0c3d-4e82-b579-fcc7ec074039 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:19:40.439866 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-c6a5466b-0c3d-4e82-b579-fcc7ec074039 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:19:40.453320 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a5466b-0c3d-4e82-b579-fcc7ec074039 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:19:40.455065 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a5466b-0c3d-4e82-b579-fcc7ec074039 service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:19:40.466287 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1456/2924] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:19:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:19:41.811419 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31336373-ed5d-45d4-9924-27b10f179556 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:19:41.813053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-31336373-ed5d-45d4-9924-27b10f179556 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:19:41.891737 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31336373-ed5d-45d4-9924-27b10f179556 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:19:41.895647 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412fca3c-4438-4957-8b76-0aab93e262a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:19:41.896337 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412fca3c-4438-4957-8b76-0aab93e262a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:19:41.896677 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412fca3c-4438-4957-8b76-0aab93e262a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:19:41.897008 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412fca3c-4438-4957-8b76-0aab93e262a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:19:41.898807 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-412fca3c-4438-4957-8b76-0aab93e262a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:19:41.912377 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412fca3c-4438-4957-8b76-0aab93e262a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:19:41.913809 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412fca3c-4438-4957-8b76-0aab93e262a0 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:19:41.925221 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1469/2925] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:19:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:19:50.461567 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:19:51.919870 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=31 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:20:33.004061 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a5466b-0c3d-4e82-b579-fcc7ec074039 service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:20:33.013258 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a5466b-0c3d-4e82-b579-fcc7ec074039 service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:20:33.016727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd37a5f7-0b92-4622-a64a-eff19f3dd8e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:20:33.017307 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd37a5f7-0b92-4622-a64a-eff19f3dd8e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:20:33.017579 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd37a5f7-0b92-4622-a64a-eff19f3dd8e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:20:33.017845 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd37a5f7-0b92-4622-a64a-eff19f3dd8e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:20:33.028363 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd37a5f7-0b92-4622-a64a-eff19f3dd8e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:20:33.029784 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd37a5f7-0b92-4622-a64a-eff19f3dd8e6 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:20:33.041666 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1457/2926] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:20:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:20:43.036454 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:21:05.937414 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412fca3c-4438-4957-8b76-0aab93e262a0 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:21:05.946638 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412fca3c-4438-4957-8b76-0aab93e262a0 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:21:05.950162 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102fee73-c900-4dc4-bfe7-48743b0a8e51 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:21:05.950891 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102fee73-c900-4dc4-bfe7-48743b0a8e51 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:21:05.951168 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102fee73-c900-4dc4-bfe7-48743b0a8e51 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:21:05.951434 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102fee73-c900-4dc4-bfe7-48743b0a8e51 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:21:05.953200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-102fee73-c900-4dc4-bfe7-48743b0a8e51 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:21:05.962461 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102fee73-c900-4dc4-bfe7-48743b0a8e51 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:21:05.963867 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102fee73-c900-4dc4-bfe7-48743b0a8e51 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:21:05.971665 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1470/2927] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:21:05 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:21:15.968078 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=6 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:22:01.816093 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd37a5f7-0b92-4622-a64a-eff19f3dd8e6 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:22:01.827212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd37a5f7-0b92-4622-a64a-eff19f3dd8e6 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:22:01.832614 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e6b3a2-0659-417a-970a-f79c5046812d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:22:01.833197 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e6b3a2-0659-417a-970a-f79c5046812d service nova] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:22:01.833472 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e6b3a2-0659-417a-970a-f79c5046812d service nova] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:22:01.833743 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e6b3a2-0659-417a-970a-f79c5046812d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:22:01.841147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e6b3a2-0659-417a-970a-f79c5046812d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:22:01.842744 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e6b3a2-0659-417a-970a-f79c5046812d service nova] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:22:01.851726 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1458/2928] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:22:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:22:11.847856 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=6 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:22:56.535851 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102fee73-c900-4dc4-bfe7-48743b0a8e51 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:22:56.537340 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-102fee73-c900-4dc4-bfe7-48743b0a8e51 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:22:56.608339 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102fee73-c900-4dc4-bfe7-48743b0a8e51 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:22:56.614494 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdfe039-f48f-4db2-90fa-76b28dff68da service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:22:56.615080 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdfe039-f48f-4db2-90fa-76b28dff68da service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:22:56.615352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdfe039-f48f-4db2-90fa-76b28dff68da service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:22:56.615617 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdfe039-f48f-4db2-90fa-76b28dff68da service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:22:56.623180 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdfe039-f48f-4db2-90fa-76b28dff68da service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:22:56.624729 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdfe039-f48f-4db2-90fa-76b28dff68da service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:22:56.633930 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1471/2929] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:22:56 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:23:06.630023 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=34 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:24:08.291722 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e6b3a2-0659-417a-970a-f79c5046812d service nova] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:24:08.350940 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e6b3a2-0659-417a-970a-f79c5046812d service nova] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:24:08.356099 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739ac13e-f427-4529-909f-bce990eadfb7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:24:08.356721 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739ac13e-f427-4529-909f-bce990eadfb7 service placement] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:24:08.357001 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739ac13e-f427-4529-909f-bce990eadfb7 service placement] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:24:08.357273 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739ac13e-f427-4529-909f-bce990eadfb7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:24:08.366636 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739ac13e-f427-4529-909f-bce990eadfb7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:24:08.368179 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739ac13e-f427-4529-909f-bce990eadfb7 service placement] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:24:08.377896 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1459/2930] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:24:08 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:24:18.373591 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=27 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:24:41.469122 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdfe039-f48f-4db2-90fa-76b28dff68da service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:24:41.488124 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdfe039-f48f-4db2-90fa-76b28dff68da service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:24:41.491726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc471a1-1023-4144-9d08-392188bc5da1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:24:41.492313 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc471a1-1023-4144-9d08-392188bc5da1 service neutron] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:24:41.492584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc471a1-1023-4144-9d08-392188bc5da1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:24:41.492852 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc471a1-1023-4144-9d08-392188bc5da1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:24:41.503498 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc471a1-1023-4144-9d08-392188bc5da1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:24:41.504857 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc471a1-1023-4144-9d08-392188bc5da1 service neutron] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:24:41.516159 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1472/2931] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:24:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:24:51.510724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=11 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:25:32.660916 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739ac13e-f427-4529-909f-bce990eadfb7 service placement] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:25:32.670569 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739ac13e-f427-4529-909f-bce990eadfb7 service placement] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:25:32.674046 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74137840-8b8c-45b4-bfcb-539bcd4c7842 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:25:32.674656 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74137840-8b8c-45b4-bfcb-539bcd4c7842 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:25:32.674937 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74137840-8b8c-45b4-bfcb-539bcd4c7842 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:25:32.675212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74137840-8b8c-45b4-bfcb-539bcd4c7842 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:25:32.676997 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-74137840-8b8c-45b4-bfcb-539bcd4c7842 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:25:32.718077 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74137840-8b8c-45b4-bfcb-539bcd4c7842 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:25:32.719517 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74137840-8b8c-45b4-bfcb-539bcd4c7842 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:25:32.739814 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1460/2932] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:25:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:25:42.737265 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=22 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:27:01.082808 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc471a1-1023-4144-9d08-392188bc5da1 service neutron] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:27:01.084411 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-8cc471a1-1023-4144-9d08-392188bc5da1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:27:01.115423 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc471a1-1023-4144-9d08-392188bc5da1 service neutron] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:27:01.120795 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3ce9e6-2df5-474d-a476-1d8af89c8961 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'dad93d99b865407aa120035015f4692a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:27:01.121382 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3ce9e6-2df5-474d-a476-1d8af89c8961 service nova] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:27:01.121649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3ce9e6-2df5-474d-a476-1d8af89c8961 service nova] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:27:01.121912 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3ce9e6-2df5-474d-a476-1d8af89c8961 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:27:01.129353 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3ce9e6-2df5-474d-a476-1d8af89c8961 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:27:01.130962 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3ce9e6-2df5-474d-a476-1d8af89c8961 service nova] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:27:01.139946 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1473/2933] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:27:01 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:27:11.136112 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=13 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:27:32.307101 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e7e0f-ae49-4186-99dc-2ae497736a3c None None] REQUEST_METHOD: `POST` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:27:32.307605 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e7e0f-ae49-4186-99dc-2ae497736a3c None None] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:27:32.307727 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e7e0f-ae49-4186-99dc-2ae497736a3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:27:32.369094 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.auth.core [None req-7a4e7e0f-ae49-4186-99dc-2ae497736a3c None None] MFA Rules not processed for user `af51bf378ea2491780250fc037fa2920`. Rule list: `[]` (Enabled: `True`). {{(pid=81432) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 02 01:27:32.370358 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7a4e7e0f-ae49-4186-99dc-2ae497736a3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:27:32.431695 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1461/2934] 10.222.0.25 () {58 vars in 967 bytes} [Sun Oct 2 01:27:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 02 01:27:32.471098 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3ce9e6-2df5-474d-a476-1d8af89c8961 service nova] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:27:32.472724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-df3ce9e6-2df5-474d-a476-1d8af89c8961 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:27:32.560459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3ce9e6-2df5-474d-a476-1d8af89c8961 service nova] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:27:32.565344 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ceb1ba-c67d-4bd8-a3fb-4b43c20f274e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'db6c9dbbcf244a6f8bac59d2c9c973bc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:27:32.565932 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ceb1ba-c67d-4bd8-a3fb-4b43c20f274e service swift] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:27:32.566239 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ceb1ba-c67d-4bd8-a3fb-4b43c20f274e service swift] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:27:32.566512 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ceb1ba-c67d-4bd8-a3fb-4b43c20f274e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:27:32.574861 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ceb1ba-c67d-4bd8-a3fb-4b43c20f274e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:27:32.576480 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ceb1ba-c67d-4bd8-a3fb-4b43c20f274e service swift] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:27:32.584632 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1474/2935] 10.222.0.25 () {60 vars in 1384 bytes} [Sun Oct 2 01:27:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 02 01:27:33.119765 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e7e0f-ae49-4186-99dc-2ae497736a3c None None] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:27:33.138516 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e7e0f-ae49-4186-99dc-2ae497736a3c None None] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:27:33.142288 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5f697-90ce-4bb8-a3cb-e7896cfa062b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:27:33.142903 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5f697-90ce-4bb8-a3cb-e7896cfa062b service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:27:33.143186 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5f697-90ce-4bb8-a3cb-e7896cfa062b service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:27:33.143456 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f5f697-90ce-4bb8-a3cb-e7896cfa062b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:27:33.145424 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-47f5f697-90ce-4bb8-a3cb-e7896cfa062b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:27:33.227154 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f5f697-90ce-4bb8-a3cb-e7896cfa062b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:27:33.229205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f5f697-90ce-4bb8-a3cb-e7896cfa062b service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:27:33.247539 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1462/2936] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:27:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:27:36.404801 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ceb1ba-c67d-4bd8-a3fb-4b43c20f274e service swift] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:27:36.421622 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ceb1ba-c67d-4bd8-a3fb-4b43c20f274e service swift] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:27:36.427459 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ccf7e1-8024-40ad-9b78-4096adc68cc4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:27:36.428352 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ccf7e1-8024-40ad-9b78-4096adc68cc4 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:27:36.428627 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ccf7e1-8024-40ad-9b78-4096adc68cc4 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:27:36.428944 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ccf7e1-8024-40ad-9b78-4096adc68cc4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:27:36.431449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-c4ccf7e1-8024-40ad-9b78-4096adc68cc4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:27:36.445095 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ccf7e1-8024-40ad-9b78-4096adc68cc4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:27:36.447026 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ccf7e1-8024-40ad-9b78-4096adc68cc4 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:27:36.467899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1475/2937] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:27:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:27:39.300237 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5f697-90ce-4bb8-a3cb-e7896cfa062b service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:27:39.301905 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-47f5f697-90ce-4bb8-a3cb-e7896cfa062b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:27:39.314467 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f5f697-90ce-4bb8-a3cb-e7896cfa062b service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:27:39.319252 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:27:39.319877 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:27:39.320156 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:27:39.320426 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] PATH_INFO: `/v3/users/47e0ae04969b4f7684cb21edaaa85b55` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:27:39.334373 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=47e0ae04969b4f7684cb21edaaa85b55)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:27:39.336296 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:27:39.338701 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=100 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:27:39.381947 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47e0ae04969b4f7684cb21edaaa85b55'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.382624 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47e0ae04969b4f7684cb21edaaa85b55'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.389394 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47e0ae04969b4f7684cb21edaaa85b55'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.390013 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47e0ae04969b4f7684cb21edaaa85b55'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.396962 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47e0ae04969b4f7684cb21edaaa85b55'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.413706 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1463/2938] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:27:39 2022] DELETE /identity/v3/users/47e0ae04969b4f7684cb21edaaa85b55 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:27:39.582593 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ccf7e1-8024-40ad-9b78-4096adc68cc4 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:27:39.593777 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ccf7e1-8024-40ad-9b78-4096adc68cc4 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:27:39.657077 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:27:39.657705 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:27:39.658636 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:27:39.658921 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] PATH_INFO: `/v3/projects/ff4a21c5f4b04110b2fa42a2bba32348` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:27:39.671258 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff4a21c5f4b04110b2fa42a2bba32348)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:27:39.673179 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:27:39.719391 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff4a21c5f4b04110b2fa42a2bba32348', 'request_id': 'req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.721044 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:27:39.728618 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff4a21c5f4b04110b2fa42a2bba32348', 'request_id': 'req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.737724 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff4a21c5f4b04110b2fa42a2bba32348', 'request_id': 'req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.748073 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] The token cache is being invalidate because project ff4a21c5f4b04110b2fa42a2bba32348 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:27:39.748408 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.748993 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.750467 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1476/2939] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:27:39 2022] DELETE /identity/v3/projects/ff4a21c5f4b04110b2fa42a2bba32348 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:27:39.754415 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:27:39.756392 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:27:39.768871 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416b3547-4dcf-44e8-aa47-9598adfb65d3 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:27:39.822741 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:27:39.823356 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:27:39.823630 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:27:39.823898 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] PATH_INFO: `/v3/users/b0c55d58b8c040479b98026cdb535e07` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:27:39.836577 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0c55d58b8c040479b98026cdb535e07)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:27:39.839007 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:27:39.882448 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0c55d58b8c040479b98026cdb535e07'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.883080 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0c55d58b8c040479b98026cdb535e07'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.889760 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0c55d58b8c040479b98026cdb535e07'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.890385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0c55d58b8c040479b98026cdb535e07'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.897088 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0c55d58b8c040479b98026cdb535e07'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:39.904385 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:27:39.917304 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1464/2940] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:27:39 2022] DELETE /identity/v3/users/b0c55d58b8c040479b98026cdb535e07 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:27:40.086522 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:27:40.088727 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:27:40.101869 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2c3028-2960-4ddf-9d2c-978a8d0e58b7 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:27:40.157596 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:27:40.158251 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:27:40.158527 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:27:40.158797 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] PATH_INFO: `/v3/projects/e64f998af7b54e848da66c621311e42b` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:27:40.168767 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e64f998af7b54e848da66c621311e42b)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:27:40.170708 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:27:40.214884 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e64f998af7b54e848da66c621311e42b', 'request_id': 'req-3d8804b1-912f-49fa-93aa-725c93924813'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:40.221766 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e64f998af7b54e848da66c621311e42b', 'request_id': 'req-3d8804b1-912f-49fa-93aa-725c93924813'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:40.228255 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e64f998af7b54e848da66c621311e42b', 'request_id': 'req-3d8804b1-912f-49fa-93aa-725c93924813'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:40.238943 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] The token cache is being invalidate because project e64f998af7b54e848da66c621311e42b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:27:40.239211 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:40.239833 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:27:40.241266 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1477/2941] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:27:40 2022] DELETE /identity/v3/projects/e64f998af7b54e848da66c621311e42b => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:27:49.913222 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=4,keystone:DELETE=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:27:50.235899 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:INSERT=2,keystone:DELETE=10,keystone:SELECT=61 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:29:11.277556 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:29:11.279033 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:29:11.358593 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08874304-5d3f-4e65-b228-b11726c6601b admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:29:11.417493 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb69081f-503a-4943-8674-6ce17eab8399 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:29:11.418123 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb69081f-503a-4943-8674-6ce17eab8399 service placement] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:29:11.418399 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb69081f-503a-4943-8674-6ce17eab8399 service placement] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:29:11.418668 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb69081f-503a-4943-8674-6ce17eab8399 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:29:11.420369 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fb69081f-503a-4943-8674-6ce17eab8399 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:29:11.431592 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb69081f-503a-4943-8674-6ce17eab8399 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:29:11.433132 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb69081f-503a-4943-8674-6ce17eab8399 service placement] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:29:11.443031 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1465/2942] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:29:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:29:21.438598 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=63 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:29:33.626573 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:29:33.627966 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:29:33.705541 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8804b1-912f-49fa-93aa-725c93924813 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:29:33.709295 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e209fb-168c-4965-beb9-45cee12de0b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'eb325d3c277b45c2a5c4e0e94078be29', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:29:33.709894 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e209fb-168c-4965-beb9-45cee12de0b3 service cinder] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:29:33.710201 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e209fb-168c-4965-beb9-45cee12de0b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:29:33.710475 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e209fb-168c-4965-beb9-45cee12de0b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:29:33.712160 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-71e209fb-168c-4965-beb9-45cee12de0b3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:29:33.790528 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e209fb-168c-4965-beb9-45cee12de0b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:29:33.791994 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e209fb-168c-4965-beb9-45cee12de0b3 service cinder] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:29:33.809510 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1478/2943] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:29:33 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:29:43.807469 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=70 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:31:42.786135 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb69081f-503a-4943-8674-6ce17eab8399 service placement] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:31:42.787620 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-fb69081f-503a-4943-8674-6ce17eab8399 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:31:42.855782 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb69081f-503a-4943-8674-6ce17eab8399 service placement] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:31:42.859212 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde20059-9ce6-40d3-86e4-ca664ca867a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e75811ee9ed7490985fed336ab9feb19', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:31:42.859806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde20059-9ce6-40d3-86e4-ca664ca867a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:31:42.860085 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde20059-9ce6-40d3-86e4-ca664ca867a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:31:42.860360 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde20059-9ce6-40d3-86e4-ca664ca867a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:31:42.862078 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-dde20059-9ce6-40d3-86e4-ca664ca867a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:31:42.923052 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde20059-9ce6-40d3-86e4-ca664ca867a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:31:42.924477 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde20059-9ce6-40d3-86e4-ca664ca867a0 service neutron] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:31:42.933738 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1466/2944] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:31:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:31:45.939720 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e209fb-168c-4965-beb9-45cee12de0b3 service cinder] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:31:45.942125 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-71e209fb-168c-4965-beb9-45cee12de0b3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:31:45.960726 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e209fb-168c-4965-beb9-45cee12de0b3 service cinder] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:31:45.965928 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af74aa3-2e76-4a05-9f07-47455aedf018 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:31:45.966671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af74aa3-2e76-4a05-9f07-47455aedf018 service placement] REQUEST_METHOD: `GET` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:31:45.967016 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af74aa3-2e76-4a05-9f07-47455aedf018 service placement] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:31:45.967349 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af74aa3-2e76-4a05-9f07-47455aedf018 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:31:45.969672 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-5af74aa3-2e76-4a05-9f07-47455aedf018 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:31:45.993479 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af74aa3-2e76-4a05-9f07-47455aedf018 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:31:45.995257 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af74aa3-2e76-4a05-9f07-47455aedf018 service placement] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:31:46.006501 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1479/2945] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:31:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:31:49.028806 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde20059-9ce6-40d3-86e4-ca664ca867a0 service neutron] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:31:49.030783 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-dde20059-9ce6-40d3-86e4-ca664ca867a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:31:49.043205 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde20059-9ce6-40d3-86e4-ca664ca867a0 service neutron] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:31:49.047873 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:31:49.048488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:31:49.048761 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:31:49.049037 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] PATH_INFO: `/v3/users/a94a5f91289c403cb49f428e53431591` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:31:49.050226 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a94a5f91289c403cb49f428e53431591)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:31:49.052097 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:31:49.096646 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a94a5f91289c403cb49f428e53431591'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.097275 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a94a5f91289c403cb49f428e53431591'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.104872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a94a5f91289c403cb49f428e53431591'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.105488 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a94a5f91289c403cb49f428e53431591'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.112723 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a94a5f91289c403cb49f428e53431591'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.129491 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1467/2946] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:31:49 2022] DELETE /identity/v3/users/a94a5f91289c403cb49f428e53431591 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:31:49.297220 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af74aa3-2e76-4a05-9f07-47455aedf018 service placement] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:31:49.307690 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af74aa3-2e76-4a05-9f07-47455aedf018 service placement] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:31:49.373018 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:31:49.373637 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:31:49.373911 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:31:49.374215 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] PATH_INFO: `/v3/projects/017c596a2d0f430d987352a616666610` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:31:49.375316 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=017c596a2d0f430d987352a616666610)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:31:49.377200 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:31:49.421977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '017c596a2d0f430d987352a616666610', 'request_id': 'req-6300a4f2-9810-4f79-accb-962cb6e93a96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.429584 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '017c596a2d0f430d987352a616666610', 'request_id': 'req-6300a4f2-9810-4f79-accb-962cb6e93a96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.435698 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '017c596a2d0f430d987352a616666610', 'request_id': 'req-6300a4f2-9810-4f79-accb-962cb6e93a96'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.445977 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] The token cache is being invalidate because project 017c596a2d0f430d987352a616666610 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:31:49.446298 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.446935 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.448931 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1480/2947] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:31:49 2022] DELETE /identity/v3/projects/017c596a2d0f430d987352a616666610 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:31:49.452975 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:31:49.454513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:31:49.468238 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3834b6f6-9c94-4298-b518-d85bd0c2fef1 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:31:49.475757 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:31:49.525203 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:31:49.525842 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:31:49.526147 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:31:49.526417 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] PATH_INFO: `/v3/users/73e4f22128c24c27ad5d3a0a48435de7` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:31:49.539177 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] RBAC: Authorizing `identity:delete_user(user_id=73e4f22128c24c27ad5d3a0a48435de7)` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:31:49.541089 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:31:49.584872 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73e4f22128c24c27ad5d3a0a48435de7'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.585513 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73e4f22128c24c27ad5d3a0a48435de7'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.592299 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73e4f22128c24c27ad5d3a0a48435de7'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.592960 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73e4f22128c24c27ad5d3a0a48435de7'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.599601 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.notifications [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73e4f22128c24c27ad5d3a0a48435de7'} {{(pid=81432) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.616698 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1468/2948] 10.222.0.25 () {58 vars in 1224 bytes} [Sun Oct 2 01:31:49 2022] DELETE /identity/v3/users/73e4f22128c24c27ad5d3a0a48435de7 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:31:49.788679 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] Authenticating user token {{(pid=81433) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:31:49.790314 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.fernet_utils [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81433) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:31:49.806542 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6300a4f2-9810-4f79-accb-962cb6e93a96 admin admin] Validating token access rules against request {{(pid=81433) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:31:49.868473 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b3c5f8b9f24f40b99d0e28c5d77526fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2ea30d4949414e4cafefc43b9507a1d0', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81433) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:31:49.869816 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:31:49.870119 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] SCRIPT_NAME: `/identity` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:31:49.870392 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] PATH_INFO: `/v3/projects/4512303a95db4c8cbf896ca76724a559` {{(pid=81433) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:31:49.880694 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4512303a95db4c8cbf896ca76724a559)` {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:31:49.882649 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] RBAC: Authorization granted {{(pid=81433) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:31:49.893111 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:31:49.931606 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4512303a95db4c8cbf896ca76724a559', 'request_id': 'req-1bfdcc68-9d97-48c3-867a-592bab8bb98a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.939671 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4512303a95db4c8cbf896ca76724a559', 'request_id': 'req-1bfdcc68-9d97-48c3-867a-592bab8bb98a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.947595 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4512303a95db4c8cbf896ca76724a559', 'request_id': 'req-1bfdcc68-9d97-48c3-867a-592bab8bb98a'} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.958053 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] The token cache is being invalidate because project 4512303a95db4c8cbf896ca76724a559 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81433) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 02 01:31:49.958362 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.958992 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG keystone.notifications [None req-1bfdcc68-9d97-48c3-867a-592bab8bb98a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81433) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 02 01:31:49.960449 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: [pid: 81433|app: 0|req: 1481/2949] 10.222.0.25 () {58 vars in 1233 bytes} [Sun Oct 2 01:31:49 2022] DELETE /identity/v3/projects/4512303a95db4c8cbf896ca76724a559 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 02 01:31:59.613327 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=53,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:31:59.955164 dv-cinder-master-3-iscsi devstack@keystone.service[81433]: DEBUG dbcounter [-] [81433] Writing DB stats keystone:SELECT=20,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81433) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 02 01:34:17.219240 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] Authenticating user token {{(pid=81432) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 02 01:34:17.220657 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:34:17.326291 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7deb7157-e78b-4a9b-a1bf-12db2e1f5080 admin admin] Validating token access rules against request {{(pid=81432) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 02 01:34:17.383195 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bdff1f-4e51-4586-ba24-729ea36fb174 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5eeff4f989ae4338a182bfdadd3a8b63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '8204e43e3cf9467cb423bb0d0f87c4fe', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81432) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 02 01:34:17.383795 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bdff1f-4e51-4586-ba24-729ea36fb174 service placement] REQUEST_METHOD: `GET` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 02 01:34:17.384069 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bdff1f-4e51-4586-ba24-729ea36fb174 service placement] SCRIPT_NAME: `/identity` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 02 01:34:17.384337 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bdff1f-4e51-4586-ba24-729ea36fb174 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81432) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 02 01:34:17.386036 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.fernet_utils [None req-34bdff1f-4e51-4586-ba24-729ea36fb174 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81432) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 02 01:34:17.397319 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bdff1f-4e51-4586-ba24-729ea36fb174 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 02 01:34:17.398929 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bdff1f-4e51-4586-ba24-729ea36fb174 service placement] RBAC: Authorization granted {{(pid=81432) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 02 01:34:17.409030 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: [pid: 81432|app: 0|req: 1469/2950] 10.222.0.25 () {60 vars in 1365 bytes} [Sun Oct 2 01:34:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 02 01:34:27.404984 dv-cinder-master-3-iscsi devstack@keystone.service[81432]: DEBUG dbcounter [-] [81432] Writing DB stats keystone:SELECT=75 {{(pid=81432) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}